Nov 03 03:00:15.723774 ip-10-0-138-120 systemd[1]: Starting Kubernetes Kubelet... Nov 03 03:00:16.128860 ip-10-0-138-120 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:00:16.128860 ip-10-0-138-120 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 03:00:16.128860 ip-10-0-138-120 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:00:16.128860 ip-10-0-138-120 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:00:16.134708 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134628 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.134708 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134707 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.134708 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134711 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134714 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134717 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134719 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134722 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134725 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134728 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134731 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134734 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134737 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134739 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134742 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134744 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134747 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134750 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134753 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134756 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134759 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134761 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.134799 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134764 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134766 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134769 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134771 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134773 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134776 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134778 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134781 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134783 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134786 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134788 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134791 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134794 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134796 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134799 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134802 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134805 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134808 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134811 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134813 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.135241 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134816 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134819 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134821 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134825 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134827 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134834 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134837 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134840 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134843 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134846 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134849 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134851 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134854 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134856 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134859 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134861 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134864 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134868 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134871 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134873 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.135719 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134876 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134878 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134881 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134884 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134886 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134891 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134894 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134896 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134899 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134902 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134904 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134906 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134909 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134912 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134914 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134916 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134919 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134921 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134924 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.136348 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134926 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134931 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134936 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134939 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134943 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134946 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134949 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134951 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134954 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134959 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134964 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134967 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134970 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134973 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134975 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134978 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134980 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134983 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134986 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.136812 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134989 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134991 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134994 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134996 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.134999 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135001 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135004 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135855 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135864 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135866 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135869 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135872 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135875 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135878 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135881 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135884 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135887 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135889 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135893 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.137250 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135895 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135898 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135901 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135904 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135906 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135909 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135911 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135914 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135916 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135919 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135921 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135924 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135929 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135932 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135934 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135937 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135939 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135942 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135944 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.137711 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135947 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135949 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135952 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135954 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135957 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135959 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135962 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135964 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135967 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135970 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135973 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135975 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135978 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135980 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135983 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135986 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135989 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135991 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135994 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135996 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.138155 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.135999 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136001 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136004 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136006 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136009 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136013 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136015 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136018 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136020 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136023 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136025 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136028 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136030 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136033 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136035 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136038 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136040 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136042 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136045 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136047 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.138653 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136050 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136052 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136054 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136059 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136065 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136068 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136071 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136074 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136077 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136079 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136082 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136084 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136087 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136089 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136092 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136094 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136096 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136100 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136102 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136105 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.139161 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136107 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136110 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136112 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136115 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136117 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136120 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136122 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136125 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136128 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136132 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136137 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136139 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136142 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136144 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136147 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136640 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136646 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136649 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136652 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.139652 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136654 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136657 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136660 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136662 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136665 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136668 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136670 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136673 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136675 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136678 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136681 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136683 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136686 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136688 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136691 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136693 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136696 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136698 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136701 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.140100 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136704 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136706 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136709 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136713 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136715 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136718 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136720 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136723 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136727 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136731 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136734 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136737 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136740 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136743 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136745 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136747 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136750 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136753 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136755 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136758 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.140552 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136760 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136763 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136765 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136768 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136770 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136773 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136775 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136778 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136780 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136783 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136785 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136788 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136791 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136794 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136796 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136800 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136802 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136805 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136808 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.141011 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136810 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136813 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136815 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136818 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136821 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136824 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136827 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136829 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136832 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136835 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136837 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136839 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136842 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136844 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136847 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136850 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136852 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136854 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136857 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136859 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.141472 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136861 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136864 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136866 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136869 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136871 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136874 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136876 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136878 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136882 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136885 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136887 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136890 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136892 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136895 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136897 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136899 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136902 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136906 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136910 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.141933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136913 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136916 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136919 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136922 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.136924 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137946 2563 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137957 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137964 2563 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137969 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137974 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137977 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137990 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137994 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.137998 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138001 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138004 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138007 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138010 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138013 2563 flags.go:64] FLAG: --cgroup-root="" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138016 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138020 2563 flags.go:64] FLAG: --client-ca-file="" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138024 2563 flags.go:64] FLAG: --cloud-provider="external" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138027 2563 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 03:00:16.142513 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138031 2563 flags.go:64] FLAG: --cluster-domain="" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138034 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138037 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138040 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138043 2563 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138047 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138050 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138053 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138057 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138060 2563 flags.go:64] FLAG: --contention-profiling="false" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138063 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138066 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138070 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138072 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138077 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138080 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138085 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138088 2563 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138091 2563 flags.go:64] FLAG: --enable-server="true" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138094 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138099 2563 flags.go:64] FLAG: --event-burst="100" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138102 2563 flags.go:64] FLAG: --event-qps="50" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138105 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138108 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 03:00:16.143050 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138110 2563 flags.go:64] FLAG: --eviction-hard="" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138115 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138118 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138121 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138124 2563 flags.go:64] FLAG: --eviction-soft="" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138126 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138131 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138134 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138136 2563 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138139 2563 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138142 2563 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138145 2563 flags.go:64] FLAG: --feature-gates="" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138150 2563 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138153 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138156 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138159 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138163 2563 flags.go:64] FLAG: --healthz-port="10248" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138167 2563 flags.go:64] FLAG: --help="false" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138170 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138173 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138176 2563 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138179 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138182 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 03:00:16.143672 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138187 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138191 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138194 2563 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138197 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138201 2563 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138203 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138207 2563 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138210 2563 flags.go:64] FLAG: --kube-reserved="" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138213 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138216 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138219 2563 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138222 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138225 2563 flags.go:64] FLAG: --lock-file="" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138227 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138230 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138234 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138239 2563 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138242 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138245 2563 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138248 2563 flags.go:64] FLAG: --logging-format="text" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138251 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138254 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138257 2563 flags.go:64] FLAG: --manifest-url="" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138260 2563 flags.go:64] FLAG: --manifest-url-header="" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138264 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 03:00:16.144215 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138267 2563 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138271 2563 flags.go:64] FLAG: --max-pods="110" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138274 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138277 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138280 2563 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138283 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138285 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138288 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138292 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138300 2563 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138303 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138305 2563 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138308 2563 flags.go:64] FLAG: --pod-cidr="" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138311 2563 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138314 2563 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138317 2563 flags.go:64] FLAG: --pods-per-core="0" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138320 2563 flags.go:64] FLAG: --port="10250" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138322 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138325 2563 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a4982af9fb55fa0e" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138328 2563 flags.go:64] FLAG: --qos-reserved="" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138331 2563 flags.go:64] FLAG: --read-only-port="10255" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138334 2563 flags.go:64] FLAG: --register-node="true" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138337 2563 flags.go:64] FLAG: --register-with-taints="" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138340 2563 flags.go:64] FLAG: --registry-burst="10" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138343 2563 flags.go:64] FLAG: --registry-qps="5" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138346 2563 flags.go:64] FLAG: --reserved-cpus="" Nov 03 03:00:16.144814 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138349 2563 flags.go:64] FLAG: --reserved-memory="" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138352 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138355 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138358 2563 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138361 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138364 2563 flags.go:64] FLAG: --runonce="false" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138366 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138369 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138372 2563 flags.go:64] FLAG: --seccomp-default="false" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138376 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138379 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138382 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138384 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138387 2563 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138391 2563 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138394 2563 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138397 2563 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138400 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138403 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138406 2563 flags.go:64] FLAG: --system-cgroups="" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138408 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138413 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138416 2563 flags.go:64] FLAG: --tls-cert-file="" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138419 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138423 2563 flags.go:64] FLAG: --tls-min-version="" Nov 03 03:00:16.145427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138425 2563 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138428 2563 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138431 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138434 2563 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138436 2563 flags.go:64] FLAG: --v="2" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138455 2563 flags.go:64] FLAG: --version="false" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138459 2563 flags.go:64] FLAG: --vmodule="" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138463 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138467 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138603 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138607 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138610 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138613 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138615 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138618 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138620 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138623 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138626 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138628 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138631 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138633 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.146024 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138637 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138639 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138642 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138645 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138647 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138650 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138652 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138655 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138657 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138660 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138663 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138666 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138669 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138671 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138674 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138676 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138678 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138681 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138683 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138686 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.146536 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138688 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138690 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138693 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138695 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138698 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138700 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138703 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138706 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138708 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138711 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138714 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138718 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138721 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138724 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138727 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138729 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138732 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138734 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138737 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.147239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138739 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138742 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138744 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138747 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138750 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138752 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138754 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138757 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138759 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138762 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138764 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138766 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138769 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138771 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138774 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138776 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138778 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138781 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138783 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138786 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.148053 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138788 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138791 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138794 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138797 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138800 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138804 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138807 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138811 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138814 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138817 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138821 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138824 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138827 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138830 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138832 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138835 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138837 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138840 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138842 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.148588 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138844 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138847 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138849 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138852 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138854 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138857 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138859 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138862 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138864 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138867 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138869 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138872 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138874 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138877 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138879 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138882 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.149038 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.138908 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138946 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138953 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138957 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138959 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138962 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138964 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138967 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138969 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138972 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138975 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138978 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138980 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138982 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138985 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138987 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138990 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138992 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138995 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138997 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.149412 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.138999 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139002 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139004 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139007 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139009 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139012 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139014 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139017 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139019 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139022 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139024 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139027 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139029 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139032 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139037 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139041 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139046 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139049 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139053 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.149906 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139056 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139059 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139062 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139064 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139067 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139069 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139072 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139074 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139077 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139079 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139082 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139084 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139086 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139089 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139092 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139094 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139096 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139099 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139101 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139104 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.150361 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139106 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139109 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139111 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139114 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139116 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139119 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139122 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139126 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139130 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139133 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139136 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139138 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139141 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139143 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139146 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139148 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139151 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139153 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139156 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139158 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.150855 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139160 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139163 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139165 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139168 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139171 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139173 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139176 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139179 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139181 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139183 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139186 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139188 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139191 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139193 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139196 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139198 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139200 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139203 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139205 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139210 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.151328 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139213 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139216 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139218 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139222 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139224 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139227 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139229 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.139232 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.148297 2563 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.148317 2563 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.148344 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.148349 2563 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148413 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148418 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148421 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148425 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148428 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148430 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148433 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.151867 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148436 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148438 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148460 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148464 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148466 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148469 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148471 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148474 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148477 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148480 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148483 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148485 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148488 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148491 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148494 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148497 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148499 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148503 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148508 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.152311 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148516 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148519 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148522 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148525 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148528 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148531 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148533 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148536 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148538 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148541 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148543 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148545 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148548 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148551 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148553 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148555 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148558 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148560 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148562 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148565 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.152770 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148567 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148570 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148573 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148575 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148577 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148580 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148583 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148585 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148588 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148590 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148593 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148596 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148600 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148604 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148608 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148611 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148614 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148617 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148619 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.153239 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148621 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148624 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148626 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148629 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148631 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148634 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148636 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148638 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148641 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148643 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148646 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148648 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148651 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148653 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148662 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148665 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148667 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148670 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148673 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148675 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.153725 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148678 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148680 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148682 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148685 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148687 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148690 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148692 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148695 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148697 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148699 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148702 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148704 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148707 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148709 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148711 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148714 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148716 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148719 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148721 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.154188 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148723 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148726 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.148754 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148898 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148902 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148905 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148908 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148910 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148920 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148923 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148925 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148928 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148930 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148933 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148935 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148938 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.154664 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148940 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148942 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148945 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148947 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148949 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148952 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148954 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148957 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148959 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148962 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148965 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148967 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148969 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148972 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148974 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148976 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148979 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148981 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148984 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.155026 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148987 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148991 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148995 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.148998 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149000 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149003 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149011 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149013 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149016 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149019 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149021 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149024 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149026 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149029 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149031 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149033 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149036 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149038 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149041 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.155478 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149044 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149046 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149049 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149052 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149054 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149057 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149059 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149062 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149064 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149066 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149069 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149071 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149074 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149076 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149078 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149081 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149083 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149086 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149088 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149091 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.155933 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149098 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149101 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149104 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149106 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149109 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149111 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149113 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149116 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149118 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149120 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149123 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149125 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149129 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149132 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149135 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149138 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149140 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149143 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149145 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.156400 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149148 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149150 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149153 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149156 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149158 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149161 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149163 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149166 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149168 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149170 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149173 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149175 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149178 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149180 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149188 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149191 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.156916 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.149215 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149246 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149250 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149253 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149255 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149258 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149260 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149263 2563 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149265 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149268 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149273 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149277 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149280 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149282 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149285 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149287 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149290 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149292 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149295 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:00:16.157294 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149297 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149300 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149303 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149305 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149307 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149310 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149312 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149315 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149318 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149320 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149323 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149325 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149333 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149336 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149339 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149341 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149344 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149346 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149348 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149351 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:00:16.157747 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149353 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149356 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149358 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149362 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149364 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149367 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149369 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149372 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149375 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149377 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149380 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149382 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149385 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149387 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149389 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149392 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149395 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149397 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149400 2563 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149402 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:00:16.158215 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149405 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149407 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149410 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149412 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149416 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149425 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149429 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149432 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149435 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149439 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149460 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149463 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149466 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149468 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149471 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149476 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149478 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149481 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149483 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:00:16.158695 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149486 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149489 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149491 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149494 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149496 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149499 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149502 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149504 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149507 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149509 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149512 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149514 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149517 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149519 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149522 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149525 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149527 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149530 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149532 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149541 2563 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:00:16.159137 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149543 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149546 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149548 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149550 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149555 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149557 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149560 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149562 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:16.149567 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.149626 2563 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.152088 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.153086 2563 server.go:1014] "Starting client certificate rotation" Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.153190 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.153233 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 03:00:16.159659 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.155373 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:00:16.160325 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.160304 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:00:16.171161 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.171140 2563 log.go:25] "Validated CRI v1 runtime API" Nov 03 03:00:16.176879 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.176861 2563 log.go:25] "Validated CRI v1 image API" Nov 03 03:00:16.177743 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.177712 2563 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 03:00:16.180610 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.180575 2563 fs.go:135] Filesystem UUIDs: map[348144bb-c90c-43ba-a724-12fe1c0d4742:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c2b12452-0a8e-44c9-ae72-ac0fe1e41a82:/dev/nvme0n1p4] Nov 03 03:00:16.180705 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.180607 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 03:00:16.186479 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.186001 2563 manager.go:217] Machine: {Timestamp:2026-11-03 03:00:16.184572365 +0000 UTC m=+0.354372415 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099476 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec281c013d75633e3b09b61cc5b3a040 SystemUUID:ec281c01-3d75-633e-3b09-b61cc5b3a040 BootID:f94219f7-05e1-4e96-837b-6354b837721a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:bb:4a:92:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:bb:4a:92:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:e3:b4:6e:23:29 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 03:00:16.187246 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.187221 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 03:00:16.187431 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.187379 2563 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 03:00:16.191377 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.191341 2563 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 03:00:16.191530 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.191377 2563 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-120.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 03:00:16.191594 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.191534 2563 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 03:00:16.191594 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.191543 2563 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 03:00:16.191650 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.191594 2563 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:00:16.191650 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.191613 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:00:16.192845 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.192834 2563 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 03:00:16.197906 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.197894 2563 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 03:00:16.198064 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.198054 2563 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 03:00:16.198872 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.198863 2563 kubelet.go:493] "Attempting to sync node with API server" Nov 03 03:00:16.198905 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.198878 2563 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 03:00:16.198905 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.198888 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 03:00:16.198905 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.198897 2563 kubelet.go:405] "Adding apiserver pod source" Nov 03 03:00:16.198905 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.198906 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 03:00:16.199256 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.199242 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:00:16.201679 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.201663 2563 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 03:00:16.208971 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.208953 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:00:16.209935 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.209919 2563 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 03:00:16.210027 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.209961 2563 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 03:00:16.210175 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210162 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 03:00:16.210238 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210182 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 03:00:16.210238 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210192 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 03:00:16.210238 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210202 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 03:00:16.210238 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210211 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 03:00:16.210238 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210222 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 03:00:16.210892 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210882 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 03:00:16.210951 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210896 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 03:00:16.210951 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210908 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 03:00:16.210951 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210917 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 03:00:16.210951 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210932 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 03:00:16.210951 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.210946 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 03:00:16.212336 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.212324 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 03:00:16.212394 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.212340 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 03:00:16.216372 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.216355 2563 server.go:1263] "Started kubelet" Nov 03 03:00:16.216482 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.216455 2563 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 03:00:16.216847 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.216622 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 03:00:16.216952 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.216933 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 03:00:16.217301 ip-10-0-138-120 systemd[1]: Started Kubernetes Kubelet. Nov 03 03:00:16.218496 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.218467 2563 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 03:00:16.219401 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.219385 2563 server.go:317] "Adding debug handlers to kubelet server" Nov 03 03:00:16.225076 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.225053 2563 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 03:00:16.226306 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.226284 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 03:00:16.226935 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.226918 2563 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 03:00:16.227148 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.227127 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:16.227922 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.227865 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 03:00:16.228723 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.228703 2563 factory.go:55] Registering systemd factory Nov 03 03:00:16.228805 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.228726 2563 factory.go:223] Registration of the systemd container factory successfully Nov 03 03:00:16.228805 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.228775 2563 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 03:00:16.228889 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.228813 2563 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 03:00:16.228983 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.228969 2563 reconstruct.go:97] "Volume reconstruction finished" Nov 03 03:00:16.229032 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.228984 2563 reconciler.go:29] "Reconciler: start to sync state" Nov 03 03:00:16.229032 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.229008 2563 factory.go:153] Registering CRI-O factory Nov 03 03:00:16.229032 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.229021 2563 factory.go:223] Registration of the crio container factory successfully Nov 03 03:00:16.229345 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.229087 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 03:00:16.229345 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.229104 2563 factory.go:103] Registering Raw factory Nov 03 03:00:16.229345 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.229129 2563 manager.go:1199] Started watching for new ooms in manager Nov 03 03:00:16.230845 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.230811 2563 manager.go:319] Starting recovery of all containers Nov 03 03:00:16.241900 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.241883 2563 manager.go:324] Recovery completed Nov 03 03:00:16.245719 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.245704 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:16.247401 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.247377 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:16.247523 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.247430 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:16.247523 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.247463 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:16.248004 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.247992 2563 cpu_manager.go:226] "Starting" policy="none" Nov 03 03:00:16.248004 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.248003 2563 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 03:00:16.248118 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.248014 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:16.248118 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.248020 2563 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 03:00:16.248742 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.248729 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:16.248851 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.248754 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:16.248851 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.248763 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:16.250137 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.250126 2563 policy_none.go:50] "Start" Nov 03 03:00:16.250169 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.250140 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 03:00:16.250169 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.250150 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 03:00:16.252158 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.252145 2563 policy_none.go:44] "Start" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.296187 2563 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.301873 2563 manager.go:341] "Starting Device Plugin manager" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.301903 2563 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.301913 2563 server.go:87] "Starting device plugin registration server" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.302203 2563 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.302215 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.302321 2563 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.302421 2563 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.302431 2563 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.303951 2563 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.304017 2563 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.314491 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.316047 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.316083 2563 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.316111 2563 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 03:00:16.319914 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.316142 2563 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 03:00:16.359869 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.359834 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-120.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 03:00:16.402698 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.402620 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:16.403727 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.403709 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:16.403858 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.403753 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:16.403858 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.403765 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:16.403858 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.403787 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.407758 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.407736 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-120.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 03:00:16.408543 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.407650 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-120.us-west-1.compute.internal.18e469bdf7236c9e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-120.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-120.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:16.216329374 +0000 UTC m=+0.386129425,LastTimestamp:2026-11-03 03:00:16.216329374 +0000 UTC m=+0.386129425,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-120.us-west-1.compute.internal,}" Nov 03 03:00:16.416743 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.416702 2563 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal"] Nov 03 03:00:16.416824 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.416782 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:16.417768 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.417750 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:16.417830 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.417784 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:16.417830 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.417798 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:16.420073 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420061 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:16.420244 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.420287 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420263 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:16.420816 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420801 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:16.420883 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420833 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:16.420883 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420845 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:16.420955 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420925 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:16.420955 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420954 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:16.421014 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.420964 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:16.423106 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.423090 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.423198 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.423117 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:16.423859 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.423841 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:16.423921 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.423870 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:16.423921 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.423880 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:16.427529 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.427503 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-120.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 03:00:16.427591 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.427503 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 03:00:16.430290 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.430274 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.430373 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.430296 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/143abffde7987a79bc89f8333af31943-config\") pod \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"143abffde7987a79bc89f8333af31943\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.430373 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.430313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.436693 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.436678 2563 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 03:00:16.436754 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.436734 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t4dwj" Nov 03 03:00:16.453926 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.453910 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" node="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.456507 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.456493 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" node="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.518320 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.518286 2563 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:16.530840 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.530815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.530933 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.530853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.530933 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.530882 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/143abffde7987a79bc89f8333af31943-config\") pod \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"143abffde7987a79bc89f8333af31943\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.531031 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.530937 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/143abffde7987a79bc89f8333af31943-config\") pod \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"143abffde7987a79bc89f8333af31943\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.531031 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.530947 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.531031 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.530937 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.618648 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.618614 2563 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:16.619109 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.619092 2563 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.619160 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.619118 2563 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-120.us-west-1.compute.internal\": node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:16.732159 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.732080 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:16.759489 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.759467 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.761634 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:16.761616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:16.832665 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.832629 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:16.933460 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:16.933417 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:17.034189 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:17.034107 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:17.135037 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:17.135002 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:17.152729 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.152694 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 03:00:17.152850 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.152827 2563 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 03:00:17.152990 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:17.152891 2563 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a058e290254ff44d893eba1331bf9de6-33cafa47f6ec0d31.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-120.us-west-1.compute.internal.18e469bdf8fe94b6\": read tcp 10.0.138.120:59270->18.221.54.233:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-120.us-west-1.compute.internal.18e469bdf8fe94b6 default 4793 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-120.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-120.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-120.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:16 +0000 UTC,LastTimestamp:2026-11-03 03:00:16.403768519 +0000 UTC m=+0.573568570,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-120.us-west-1.compute.internal,}" Nov 03 03:00:17.235113 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:17.235067 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:17.336078 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:17.336004 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:17.436750 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:17.436712 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:00:17.442889 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.442872 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:00:17.442967 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.442862 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 02:55:16 +0000 UTC" deadline="2028-05-26 03:06:36.277792277 +0000 UTC" Nov 03 03:00:17.442967 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.442960 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13680h6m18.834834947s" Nov 03 03:00:17.444006 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.443989 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:00:17.444096 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.444021 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:00:17.444096 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.444036 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:00:17.513154 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.513116 2563 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:17.528015 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.527977 2563 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:17.576623 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.576592 2563 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:17.604163 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.604103 2563 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:00:17.605049 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.605034 2563 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:00:17.667149 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:17.667119 2563 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:00:17.997990 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:17.997947 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fdaf3bc04b642a1b5c4542f0b4daec.slice/crio-7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2 WatchSource:0}: Error finding container 7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2: Status 404 returned error can't find the container with id 7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2 Nov 03 03:00:17.998422 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:17.998397 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143abffde7987a79bc89f8333af31943.slice/crio-6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222 WatchSource:0}: Error finding container 6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222: Status 404 returned error can't find the container with id 6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222 Nov 03 03:00:18.003828 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.003814 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:00:18.200193 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.200104 2563 apiserver.go:52] "Watching apiserver" Nov 03 03:00:18.227319 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.227288 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 03:00:18.320993 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.320930 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"e9fdaf3bc04b642a1b5c4542f0b4daec","Type":"ContainerStarted","Data":"7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2"} Nov 03 03:00:18.321885 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.321866 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"143abffde7987a79bc89f8333af31943","Type":"ContainerStarted","Data":"6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222"} Nov 03 03:00:18.342158 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.342120 2563 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 03:00:18.342426 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.342409 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2fd9d","openshift-ovn-kubernetes/ovnkube-node-7glvk","kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-pmq44","openshift-image-registry/node-ca-vnrpn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal","openshift-dns/node-resolver-dtzkv","openshift-multus/network-metrics-daemon-xw724","kube-system/konnectivity-agent-qg9cd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l","kube-system/global-pull-secret-syncer-j76qm","openshift-multus/multus-additional-cni-plugins-hzvq6","openshift-multus/multus-ssswp","openshift-network-diagnostics/network-check-target-ldf8b"] Nov 03 03:00:18.345909 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.345892 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.347835 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.347808 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.349730 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.349712 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:18.349813 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.349768 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:18.351816 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.351677 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.353723 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.353703 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.355675 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.355654 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:18.355746 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.355723 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:18.357668 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.357652 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.359687 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.359672 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.362487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.362340 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.364669 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.364650 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.367510 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.367490 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.367592 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.367562 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:18.369659 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.369645 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.393957 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.393932 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 03:00:18.394149 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.394132 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcpl5\"" Nov 03 03:00:18.429355 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.429324 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 03:00:18.430426 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.430410 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgzd9\"" Nov 03 03:00:18.430719 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.430694 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tvk4\"" Nov 03 03:00:18.431021 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.431009 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrqj7\"" Nov 03 03:00:18.431302 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.431286 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 03:00:18.431302 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.431296 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 03:00:18.434302 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.434284 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 03:00:18.434371 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.434344 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 03:00:18.435908 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.435890 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:18.436152 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436139 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 03:00:18.436211 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436182 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:18.436211 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436201 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 03:00:18.436288 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436206 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 03:00:18.436288 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436219 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:18.436365 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436344 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 03:00:18.436365 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436356 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:18.436476 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436413 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 03:00:18.436476 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436463 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 03:00:18.436567 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436520 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 03:00:18.436567 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436531 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 03:00:18.436567 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436523 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 03:00:18.436567 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436555 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 03:00:18.436567 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436562 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 03:00:18.436754 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436585 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 03:00:18.436754 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.436600 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 03:00:18.439463 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.439423 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fg6rm\"" Nov 03 03:00:18.441720 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441703 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b70536d4-1f3e-456d-9c5c-63f25de2a77f-hosts-file\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.441793 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441735 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-multus-certs\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.441845 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441800 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-etc-kubernetes\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.441845 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441832 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/517134a0-0ad0-46d8-a99b-ed28cce40ae5-host\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.441927 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-log-socket\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.441927 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-cnibin\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.442005 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441932 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztvl\" (UniqueName: \"kubernetes.io/projected/b70536d4-1f3e-456d-9c5c-63f25de2a77f-kube-api-access-fztvl\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.442005 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441958 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-etc-tuned\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.442005 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xgb\" (UniqueName: \"kubernetes.io/projected/1714a90e-d9bd-4a76-bc61-295f6978bf17-kube-api-access-d2xgb\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:18.442005 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.441998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-bin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.442132 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.442132 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442049 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovn-node-metrics-cert\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442132 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-script-lib\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442132 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442104 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-dbus\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.442132 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442120 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cni-binary-copy\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442226 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-netns\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442274 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-multus\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442303 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-hostroot\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442335 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-device-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442358 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-sys-fs\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.442376 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/724f0110-6fa5-473c-a88b-c514952a50a3-konnectivity-ca\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442397 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gx4h\" (UniqueName: \"kubernetes.io/projected/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-kube-api-access-6gx4h\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-systemd-units\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-slash\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442492 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-systemd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442514 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-env-overrides\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442537 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-sys\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442570 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-var-lib-kubelet\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-host\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442621 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442651 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/724f0110-6fa5-473c-a88b-c514952a50a3-agent-certs\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442678 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-var-lib-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-etc-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-config\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.442781 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-registration-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-bin\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-conf-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442849 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmswh\" (UniqueName: \"kubernetes.io/projected/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-kube-api-access-hmswh\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442873 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysconfig\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442916 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgq6x\" (UniqueName: \"kubernetes.io/projected/fb498359-0464-4428-8a7d-4901cea017ac-kube-api-access-tgq6x\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442943 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-kubelet\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-system-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.442978 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70536d4-1f3e-456d-9c5c-63f25de2a77f-tmp-dir\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-modprobe-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-lib-modules\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443071 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-host-slash\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/517134a0-0ad0-46d8-a99b-ed28cce40ae5-serviceca\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443121 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-system-cni-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443156 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-k8s-cni-cncf-io\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443181 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-daemon-config\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443218 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-conf\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443245 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-tmp\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443285 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443323 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-kubelet-config\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443378 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443404 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn5q2\" (UniqueName: \"kubernetes.io/projected/517134a0-0ad0-46d8-a99b-ed28cce40ae5-kube-api-access-dn5q2\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-netd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443437 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-socket-dir-parent\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443461 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-kubelet\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443474 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-systemd\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443487 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-run\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-ovn\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443514 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pqmg\" (UniqueName: \"kubernetes.io/projected/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-kube-api-access-9pqmg\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.443785 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443527 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnf6\" (UniqueName: \"kubernetes.io/projected/19f60386-016f-4213-ad61-462deca38b28-kube-api-access-stnf6\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443539 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-kubernetes\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443553 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-iptables-alerter-script\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-netns\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443622 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-socket-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443682 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgnt\" (UniqueName: \"kubernetes.io/projected/57dbddca-a044-4478-8cfe-9867494e4291-kube-api-access-dsgnt\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cnibin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443720 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443738 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-node-log\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.444182 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.443760 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.476196 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.476123 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fp5b4\"" Nov 03 03:00:18.476196 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.476158 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rnqls\"" Nov 03 03:00:18.476536 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.476520 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d2r77\"" Nov 03 03:00:18.476591 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.476554 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 03:00:18.479769 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.479752 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 03:00:18.479821 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.479814 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 03:00:18.479903 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.479892 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqjzv\"" Nov 03 03:00:18.481243 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.481231 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 03:00:18.481369 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.481357 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 03:00:18.481407 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.481387 2563 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 03:00:18.481474 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.481461 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z6k9f" Nov 03 03:00:18.488484 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.488467 2563 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:00:18.544568 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-netns\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.544568 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-multus\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544593 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-hostroot\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544609 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-netns\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544616 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-device-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544632 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-multus\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544649 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-hostroot\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-sys-fs\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544688 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/724f0110-6fa5-473c-a88b-c514952a50a3-konnectivity-ca\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544712 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-sys-fs\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gx4h\" (UniqueName: \"kubernetes.io/projected/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-kube-api-access-6gx4h\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.544752 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544689 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-device-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544760 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-systemd-units\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-systemd-units\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544793 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-slash\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544827 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-slash\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-systemd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-env-overrides\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544887 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-sys\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-var-lib-kubelet\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-host\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544949 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544964 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-sys\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/724f0110-6fa5-473c-a88b-c514952a50a3-agent-certs\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544982 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-host\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-var-lib-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.544987 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-systemd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-etc-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545024 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-var-lib-kubelet\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545050 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-var-lib-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545055 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-etc-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.545059 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-config\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.545137 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:19.045103073 +0000 UTC m=+3.214903136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545162 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-registration-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-bin\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545200 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-conf-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545202 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/724f0110-6fa5-473c-a88b-c514952a50a3-konnectivity-ca\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmswh\" (UniqueName: \"kubernetes.io/projected/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-kube-api-access-hmswh\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545236 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-registration-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545236 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-bin\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545261 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-conf-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545279 2563 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545315 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysconfig\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysconfig\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545327 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-env-overrides\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.545854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545370 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq6x\" (UniqueName: \"kubernetes.io/projected/fb498359-0464-4428-8a7d-4901cea017ac-kube-api-access-tgq6x\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-kubelet\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545418 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-system-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545462 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70536d4-1f3e-456d-9c5c-63f25de2a77f-tmp-dir\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545467 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-kubelet\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-modprobe-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545489 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-system-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-lib-modules\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545546 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-host-slash\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545564 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-config\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/517134a0-0ad0-46d8-a99b-ed28cce40ae5-serviceca\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545597 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-modprobe-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545607 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-host-slash\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-system-cni-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-k8s-cni-cncf-io\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545656 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-system-cni-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545668 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-daemon-config\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545680 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-lib-modules\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.546637 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545697 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-k8s-cni-cncf-io\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-conf\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545723 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70536d4-1f3e-456d-9c5c-63f25de2a77f-tmp-dir\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545728 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-tmp\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545776 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-kubelet-config\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545846 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545894 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-kubelet-config\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn5q2\" (UniqueName: \"kubernetes.io/projected/517134a0-0ad0-46d8-a99b-ed28cce40ae5-kube-api-access-dn5q2\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545959 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545846 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-conf\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-netd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.547471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545990 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545992 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-netd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546006 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/517134a0-0ad0-46d8-a99b-ed28cce40ae5-serviceca\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.545875 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546007 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-socket-dir-parent\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546055 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-kubelet\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-systemd\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-run\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546116 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-daemon-config\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546122 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-ovn\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmg\" (UniqueName: \"kubernetes.io/projected/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-kube-api-access-9pqmg\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnf6\" (UniqueName: \"kubernetes.io/projected/19f60386-016f-4213-ad61-462deca38b28-kube-api-access-stnf6\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546187 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-systemd\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546191 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-kubernetes\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-iptables-alerter-script\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546235 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-socket-dir-parent\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546247 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-netns\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546235 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-ovn\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546272 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-kubernetes\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-socket-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546298 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-run\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546319 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-netns\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546333 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546425 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-socket-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546471 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546480 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-kubelet\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546510 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgnt\" (UniqueName: \"kubernetes.io/projected/57dbddca-a044-4478-8cfe-9867494e4291-kube-api-access-dsgnt\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cnibin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-node-log\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546769 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b70536d4-1f3e-456d-9c5c-63f25de2a77f-hosts-file\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-multus-certs\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548521 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546830 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-etc-kubernetes\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546850 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/517134a0-0ad0-46d8-a99b-ed28cce40ae5-host\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546869 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-log-socket\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546875 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-node-log\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546883 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-iptables-alerter-script\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-cnibin\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546924 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-cnibin\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztvl\" (UniqueName: \"kubernetes.io/projected/b70536d4-1f3e-456d-9c5c-63f25de2a77f-kube-api-access-fztvl\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546950 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-etc-tuned\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546980 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b70536d4-1f3e-456d-9c5c-63f25de2a77f-hosts-file\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xgb\" (UniqueName: \"kubernetes.io/projected/1714a90e-d9bd-4a76-bc61-295f6978bf17-kube-api-access-d2xgb\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547012 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/517134a0-0ad0-46d8-a99b-ed28cce40ae5-host\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547021 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-log-socket\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547014 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-bin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547034 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-bin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547054 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cnibin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547057 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.548924 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.546963 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547082 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovn-node-metrics-cert\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547083 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-multus-certs\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547104 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-etc-kubernetes\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-script-lib\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-dbus\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547166 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547233 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-dbus\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.547282 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547303 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cni-binary-copy\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.547353 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret podName:68969cec-a620-4bd5-bfd1-cd203d0822b8 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:19.047319371 +0000 UTC m=+3.217119407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret") pod "global-pull-secret-syncer-j76qm" (UID: "68969cec-a620-4bd5-bfd1-cd203d0822b8") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.547658 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-script-lib\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.548300 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.548410 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cni-binary-copy\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.549478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.548486 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.549937 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.548929 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-tmp\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.549937 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.549408 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-etc-tuned\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.549937 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.549464 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/724f0110-6fa5-473c-a88b-c514952a50a3-agent-certs\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.549937 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.549484 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovn-node-metrics-cert\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.602635 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.602600 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgnt\" (UniqueName: \"kubernetes.io/projected/57dbddca-a044-4478-8cfe-9867494e4291-kube-api-access-dsgnt\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.606840 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.606810 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:18.606840 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.606834 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:18.606840 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.606848 2563 projected.go:196] Error preparing data for projected volume kube-api-access-r4k7s for pod openshift-network-diagnostics/network-check-target-ldf8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:18.607060 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:18.606910 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s podName:4b8aed8e-c303-437e-9e6e-37d9334a1541 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:19.106892228 +0000 UTC m=+3.276692284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r4k7s" (UniqueName: "kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s") pod "network-check-target-ldf8b" (UID: "4b8aed8e-c303-437e-9e6e-37d9334a1541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:18.609236 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.609206 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq6x\" (UniqueName: \"kubernetes.io/projected/fb498359-0464-4428-8a7d-4901cea017ac-kube-api-access-tgq6x\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.609606 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.609586 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn5q2\" (UniqueName: \"kubernetes.io/projected/517134a0-0ad0-46d8-a99b-ed28cce40ae5-kube-api-access-dn5q2\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.609663 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.609587 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztvl\" (UniqueName: \"kubernetes.io/projected/b70536d4-1f3e-456d-9c5c-63f25de2a77f-kube-api-access-fztvl\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.609692 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.609654 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xgb\" (UniqueName: \"kubernetes.io/projected/1714a90e-d9bd-4a76-bc61-295f6978bf17-kube-api-access-d2xgb\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:18.610187 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.610165 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmswh\" (UniqueName: \"kubernetes.io/projected/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-kube-api-access-hmswh\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.610527 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.610510 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnf6\" (UniqueName: \"kubernetes.io/projected/19f60386-016f-4213-ad61-462deca38b28-kube-api-access-stnf6\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.610583 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.610522 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqmg\" (UniqueName: \"kubernetes.io/projected/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-kube-api-access-9pqmg\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.613909 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.613895 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gx4h\" (UniqueName: \"kubernetes.io/projected/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-kube-api-access-6gx4h\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.656937 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.656909 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:00:18.661883 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.661864 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ssswp" Nov 03 03:00:18.666154 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.666131 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70536d4_1f3e_456d_9c5c_63f25de2a77f.slice/crio-4070fa999057d8bd19e16e519137e585ff5901a16e261fb86c2b7a946b82aa47 WatchSource:0}: Error finding container 4070fa999057d8bd19e16e519137e585ff5901a16e261fb86c2b7a946b82aa47: Status 404 returned error can't find the container with id 4070fa999057d8bd19e16e519137e585ff5901a16e261fb86c2b7a946b82aa47 Nov 03 03:00:18.668910 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.668894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:00:18.670198 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.670177 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c75a11d_6c39_4d24_ac43_0b0dc04f4faf.slice/crio-41c27335458a2d17f09e3bef82fe1b19455ea376bc7130e1a04bc97477ee9acd WatchSource:0}: Error finding container 41c27335458a2d17f09e3bef82fe1b19455ea376bc7130e1a04bc97477ee9acd: Status 404 returned error can't find the container with id 41c27335458a2d17f09e3bef82fe1b19455ea376bc7130e1a04bc97477ee9acd Nov 03 03:00:18.671813 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.671794 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:00:18.677329 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.677309 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:18.678035 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.678005 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb498359_0464_4428_8a7d_4901cea017ac.slice/crio-9345036417c4c5fc65c3c7c2ff40d8e70d2afe72317b6cb89aecf8af98a8a26f WatchSource:0}: Error finding container 9345036417c4c5fc65c3c7c2ff40d8e70d2afe72317b6cb89aecf8af98a8a26f: Status 404 returned error can't find the container with id 9345036417c4c5fc65c3c7c2ff40d8e70d2afe72317b6cb89aecf8af98a8a26f Nov 03 03:00:18.679964 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.679945 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517134a0_0ad0_46d8_a99b_ed28cce40ae5.slice/crio-4267a533b031d43686708ddfebe0b727a362020cbaa383c778e174b715ab1dc6 WatchSource:0}: Error finding container 4267a533b031d43686708ddfebe0b727a362020cbaa383c778e174b715ab1dc6: Status 404 returned error can't find the container with id 4267a533b031d43686708ddfebe0b727a362020cbaa383c778e174b715ab1dc6 Nov 03 03:00:18.683320 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.683303 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:00:18.693172 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.693141 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb3b873_057e_44cd_9ec6_f22dd6f809b2.slice/crio-0a3f9e980982f175a5aef7c89876a2d31fc55c56e894f09cbd8f67e57bbb737a WatchSource:0}: Error finding container 0a3f9e980982f175a5aef7c89876a2d31fc55c56e894f09cbd8f67e57bbb737a: Status 404 returned error can't find the container with id 0a3f9e980982f175a5aef7c89876a2d31fc55c56e894f09cbd8f67e57bbb737a Nov 03 03:00:18.707843 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.707820 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:00:18.712419 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.712391 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:18.718041 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.718020 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f60386_016f_4213_ad61_462deca38b28.slice/crio-fed7ab78d88bd0b4128b19529522d09ef1b960ab09e1046d7e2b14577bbf8584 WatchSource:0}: Error finding container fed7ab78d88bd0b4128b19529522d09ef1b960ab09e1046d7e2b14577bbf8584: Status 404 returned error can't find the container with id fed7ab78d88bd0b4128b19529522d09ef1b960ab09e1046d7e2b14577bbf8584 Nov 03 03:00:18.720524 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:18.720426 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:00:18.720791 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.720769 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b47370a_1fb8_43ef_8ecb_c0ff2eeb56af.slice/crio-a9acee61234ba09bec1e361ee402c94081657d7280744d288f8140a29ed1f6c6 WatchSource:0}: Error finding container a9acee61234ba09bec1e361ee402c94081657d7280744d288f8140a29ed1f6c6: Status 404 returned error can't find the container with id a9acee61234ba09bec1e361ee402c94081657d7280744d288f8140a29ed1f6c6 Nov 03 03:00:18.729634 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:18.729614 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dbddca_a044_4478_8cfe_9867494e4291.slice/crio-722128c275ce379f3503e2b3fe3cf26bf0d3a0b451e8b7570bbed3c240c7a593 WatchSource:0}: Error finding container 722128c275ce379f3503e2b3fe3cf26bf0d3a0b451e8b7570bbed3c240c7a593: Status 404 returned error can't find the container with id 722128c275ce379f3503e2b3fe3cf26bf0d3a0b451e8b7570bbed3c240c7a593 Nov 03 03:00:19.051583 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.051494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:19.051583 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.051551 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:19.051761 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.051656 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:19.051761 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.051687 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:19.051761 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.051723 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret podName:68969cec-a620-4bd5-bfd1-cd203d0822b8 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:20.051704614 +0000 UTC m=+4.221504664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret") pod "global-pull-secret-syncer-j76qm" (UID: "68969cec-a620-4bd5-bfd1-cd203d0822b8") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:19.051761 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.051740 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:20.051733128 +0000 UTC m=+4.221533163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:19.152197 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.152168 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:19.152391 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.152371 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:19.152462 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.152399 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:19.152462 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.152412 2563 projected.go:196] Error preparing data for projected volume kube-api-access-r4k7s for pod openshift-network-diagnostics/network-check-target-ldf8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:19.152549 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:19.152523 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s podName:4b8aed8e-c303-437e-9e6e-37d9334a1541 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:20.152505328 +0000 UTC m=+4.322305379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4k7s" (UniqueName: "kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s") pod "network-check-target-ldf8b" (UID: "4b8aed8e-c303-437e-9e6e-37d9334a1541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:19.325163 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.325076 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qg9cd" event={"ID":"724f0110-6fa5-473c-a88b-c514952a50a3","Type":"ContainerStarted","Data":"31bb7507a68daa485ad5e88fe5d112729bba7534e1d0a79c4c57e2ddbbe581be"} Nov 03 03:00:19.326593 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.326534 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ssswp" event={"ID":"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf","Type":"ContainerStarted","Data":"41c27335458a2d17f09e3bef82fe1b19455ea376bc7130e1a04bc97477ee9acd"} Nov 03 03:00:19.327761 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.327734 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtzkv" event={"ID":"b70536d4-1f3e-456d-9c5c-63f25de2a77f","Type":"ContainerStarted","Data":"4070fa999057d8bd19e16e519137e585ff5901a16e261fb86c2b7a946b82aa47"} Nov 03 03:00:19.329145 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.329118 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"a9acee61234ba09bec1e361ee402c94081657d7280744d288f8140a29ed1f6c6"} Nov 03 03:00:19.330373 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.330345 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fd9d" event={"ID":"0eb3b873-057e-44cd-9ec6-f22dd6f809b2","Type":"ContainerStarted","Data":"0a3f9e980982f175a5aef7c89876a2d31fc55c56e894f09cbd8f67e57bbb737a"} Nov 03 03:00:19.331555 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.331530 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnrpn" event={"ID":"517134a0-0ad0-46d8-a99b-ed28cce40ae5","Type":"ContainerStarted","Data":"4267a533b031d43686708ddfebe0b727a362020cbaa383c778e174b715ab1dc6"} Nov 03 03:00:19.332656 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.332620 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmq44" event={"ID":"fb498359-0464-4428-8a7d-4901cea017ac","Type":"ContainerStarted","Data":"9345036417c4c5fc65c3c7c2ff40d8e70d2afe72317b6cb89aecf8af98a8a26f"} Nov 03 03:00:19.333748 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.333719 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerStarted","Data":"722128c275ce379f3503e2b3fe3cf26bf0d3a0b451e8b7570bbed3c240c7a593"} Nov 03 03:00:19.334870 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.334847 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" event={"ID":"19f60386-016f-4213-ad61-462deca38b28","Type":"ContainerStarted","Data":"fed7ab78d88bd0b4128b19529522d09ef1b960ab09e1046d7e2b14577bbf8584"} Nov 03 03:00:19.484005 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.483967 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:18 +0000 UTC" deadline="2028-06-02 09:05:17.719157007 +0000 UTC" Nov 03 03:00:19.484005 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:19.484003 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13854h4m58.235158558s" Nov 03 03:00:20.059593 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.059549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:20.059771 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.059602 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:20.059771 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.059721 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:20.059871 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.059784 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:22.059765084 +0000 UTC m=+6.229565122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:20.059871 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.059844 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:20.059976 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.059875 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret podName:68969cec-a620-4bd5-bfd1-cd203d0822b8 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:22.05986522 +0000 UTC m=+6.229665258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret") pod "global-pull-secret-syncer-j76qm" (UID: "68969cec-a620-4bd5-bfd1-cd203d0822b8") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:20.160268 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.160230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:20.160514 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.160397 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:20.160514 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.160424 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:20.160514 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.160435 2563 projected.go:196] Error preparing data for projected volume kube-api-access-r4k7s for pod openshift-network-diagnostics/network-check-target-ldf8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:20.160695 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.160539 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s podName:4b8aed8e-c303-437e-9e6e-37d9334a1541 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:22.160522374 +0000 UTC m=+6.330322423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4k7s" (UniqueName: "kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s") pod "network-check-target-ldf8b" (UID: "4b8aed8e-c303-437e-9e6e-37d9334a1541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:20.322899 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.322581 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:20.322899 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.322698 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:20.322899 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.322817 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:20.322899 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.322825 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:20.323280 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.322931 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:20.323280 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:20.323012 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:20.484455 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.484338 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:18 +0000 UTC" deadline="2028-05-25 07:48:12.413137522 +0000 UTC" Nov 03 03:00:20.484455 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:20.484401 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13660h47m51.928740501s" Nov 03 03:00:21.352540 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:21.352501 2563 generic.go:356] "Generic (PLEG): container finished" podID="e9fdaf3bc04b642a1b5c4542f0b4daec" containerID="7abcaa7899297e64a6b03d70e15ea464ba9f313c50caaeaabf859fae0c58c94e" exitCode=0 Nov 03 03:00:21.352704 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:21.352608 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"e9fdaf3bc04b642a1b5c4542f0b4daec","Type":"ContainerDied","Data":"7abcaa7899297e64a6b03d70e15ea464ba9f313c50caaeaabf859fae0c58c94e"} Nov 03 03:00:21.367766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:21.367701 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"143abffde7987a79bc89f8333af31943","Type":"ContainerStarted","Data":"b7a1dcf474e5d04e386d0f66bf30e2920d4c8005e8bfca678f31f3c46fedbe0f"} Nov 03 03:00:21.589888 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:21.589813 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" podStartSLOduration=4.589793736 podStartE2EDuration="4.589793736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:21.588649593 +0000 UTC m=+5.758449677" watchObservedRunningTime="2026-11-03 03:00:21.589793736 +0000 UTC m=+5.759593790" Nov 03 03:00:22.076238 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.075772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:22.076238 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.075828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:22.076238 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.075978 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:22.076238 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.076036 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:26.076018021 +0000 UTC m=+10.245818063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:22.076238 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.076113 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:22.076238 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.076146 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret podName:68969cec-a620-4bd5-bfd1-cd203d0822b8 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:26.0761359 +0000 UTC m=+10.245935938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret") pod "global-pull-secret-syncer-j76qm" (UID: "68969cec-a620-4bd5-bfd1-cd203d0822b8") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:22.177128 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.176892 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:22.177128 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.177016 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:22.177128 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.177031 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:22.177128 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.177039 2563 projected.go:196] Error preparing data for projected volume kube-api-access-r4k7s for pod openshift-network-diagnostics/network-check-target-ldf8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:22.177128 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.177098 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s podName:4b8aed8e-c303-437e-9e6e-37d9334a1541 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:26.177079109 +0000 UTC m=+10.346879170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4k7s" (UniqueName: "kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s") pod "network-check-target-ldf8b" (UID: "4b8aed8e-c303-437e-9e6e-37d9334a1541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:22.318586 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.318552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:22.318733 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.318669 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:22.318733 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.318725 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:22.318815 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.318770 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:22.318815 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.318807 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:22.318875 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:22.318854 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:22.378817 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.377803 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"e9fdaf3bc04b642a1b5c4542f0b4daec","Type":"ContainerStarted","Data":"7138db1a7786853dd4cd7a0c491c538ce539589d8451598d5d0e97987106184b"} Nov 03 03:00:22.383832 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.383796 2563 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="6a8cc15861287e2bc932f28c40da9c9d972ed37cf717f39668aca59db4cb8fb2" exitCode=0 Nov 03 03:00:22.383958 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.383881 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerDied","Data":"6a8cc15861287e2bc932f28c40da9c9d972ed37cf717f39668aca59db4cb8fb2"} Nov 03 03:00:22.619788 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:22.619727 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" podStartSLOduration=5.619705913 podStartE2EDuration="5.619705913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:22.495936837 +0000 UTC m=+6.665736914" watchObservedRunningTime="2026-11-03 03:00:22.619705913 +0000 UTC m=+6.789505974" Nov 03 03:00:24.320055 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:24.319623 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:24.320055 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:24.319767 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:24.320055 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:24.319870 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:24.320055 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:24.319990 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:24.320717 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:24.320071 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:24.320717 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:24.320259 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:24.391841 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:24.391088 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnrpn" event={"ID":"517134a0-0ad0-46d8-a99b-ed28cce40ae5","Type":"ContainerStarted","Data":"c9fd7a74b1849d3c24a529b7aabcbecadab299d7a8c0ad1621c6cfdfaaa9be58"} Nov 03 03:00:26.115033 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.114990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:26.115550 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.115050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:26.115550 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.115194 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:26.115550 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.115262 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:34.115244154 +0000 UTC m=+18.285044193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:26.115550 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.115331 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:26.115550 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.115369 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret podName:68969cec-a620-4bd5-bfd1-cd203d0822b8 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:34.115356709 +0000 UTC m=+18.285156749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret") pod "global-pull-secret-syncer-j76qm" (UID: "68969cec-a620-4bd5-bfd1-cd203d0822b8") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:26.216097 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.216054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:26.216245 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.216226 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:26.216245 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.216245 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:26.216351 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.216258 2563 projected.go:196] Error preparing data for projected volume kube-api-access-r4k7s for pod openshift-network-diagnostics/network-check-target-ldf8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:26.216351 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.216317 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s podName:4b8aed8e-c303-437e-9e6e-37d9334a1541 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:34.216298058 +0000 UTC m=+18.386098113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4k7s" (UniqueName: "kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s") pod "network-check-target-ldf8b" (UID: "4b8aed8e-c303-437e-9e6e-37d9334a1541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:26.319473 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.319429 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:26.319633 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.319568 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:26.319633 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.319608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:26.319757 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.319678 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:26.319757 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.319714 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:26.319856 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:26.319776 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:26.396908 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.396850 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ssswp" event={"ID":"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf","Type":"ContainerStarted","Data":"48c603b89e5c53a4afb4faf3aefe14b17fe4d67fd9f4561bfda69e0bb83c4b88"} Nov 03 03:00:26.398772 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.398735 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtzkv" event={"ID":"b70536d4-1f3e-456d-9c5c-63f25de2a77f","Type":"ContainerStarted","Data":"f0f3f39e70c066133515d7645b31a26bda6c87634428f9f87d4aed6c161906c9"} Nov 03 03:00:26.516053 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.515950 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vnrpn" podStartSLOduration=5.534279873 podStartE2EDuration="10.515929594s" totalImagesPullingTime="4.981649721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:24.508208608 +0000 UTC m=+8.678008669" watchObservedRunningTime="2026-11-03 03:00:26.515929594 +0000 UTC m=+10.685729655" Nov 03 03:00:26.516502 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:26.516467 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ssswp" podStartSLOduration=3.787773621 podStartE2EDuration="10.516458168s" totalImagesPullingTime="6.728684547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:26.515918412 +0000 UTC m=+10.685718475" watchObservedRunningTime="2026-11-03 03:00:26.516458168 +0000 UTC m=+10.686258228" Nov 03 03:00:28.317375 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.317283 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:28.317375 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.317322 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:28.317375 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.317332 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:28.317850 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:28.317435 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:28.317850 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:28.317564 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:28.317850 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:28.317679 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:28.405037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.404998 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fd9d" event={"ID":"0eb3b873-057e-44cd-9ec6-f22dd6f809b2","Type":"ContainerStarted","Data":"5c3b6e402ddb2a9f38ea168e89326f6de15a3059016af84eee2837dfe2802d24"} Nov 03 03:00:28.406997 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.406965 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmq44" event={"ID":"fb498359-0464-4428-8a7d-4901cea017ac","Type":"ContainerStarted","Data":"38daf16efed56c9fb243a8e4611ce938236ca01f53a4bc401fa6dc4e42d0ab72"} Nov 03 03:00:28.529341 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.529286 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2fd9d" podStartSLOduration=5.2931557940000005 podStartE2EDuration="12.529268178s" totalImagesPullingTime="7.236112384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:28.528941195 +0000 UTC m=+12.698741256" watchObservedRunningTime="2026-11-03 03:00:28.529268178 +0000 UTC m=+12.699068235" Nov 03 03:00:28.529780 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.529751 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dtzkv" podStartSLOduration=5.162788435 podStartE2EDuration="12.529739968s" totalImagesPullingTime="7.366951533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:26.63256307 +0000 UTC m=+10.802363130" watchObservedRunningTime="2026-11-03 03:00:28.529739968 +0000 UTC m=+12.699540434" Nov 03 03:00:28.647489 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:28.647365 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pmq44" podStartSLOduration=3.344837522 podStartE2EDuration="12.647348955s" totalImagesPullingTime="9.302511433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:28.646380276 +0000 UTC m=+12.816180339" watchObservedRunningTime="2026-11-03 03:00:28.647348955 +0000 UTC m=+12.817149016" Nov 03 03:00:30.316987 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:30.316945 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:30.317413 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:30.316949 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:30.317413 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:30.317103 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:30.317413 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:30.317113 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:30.317413 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:30.317209 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:30.317413 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:30.317297 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:32.316888 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.316723 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:32.317618 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.316723 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:32.317618 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:32.316996 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:32.317618 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.316758 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:32.317618 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:32.317054 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:32.317618 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:32.317130 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:32.418948 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.418917 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"1ed64d6f28cfefb9cafe848e5947add20ac59b9176afe87580c4b57af3e4eda5"} Nov 03 03:00:32.419095 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.418993 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"2326ec47a2c07a51f6f4db4fd42db4e126d44cbcfc34c10cf09a5b2cfd70f719"} Nov 03 03:00:32.419095 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.419023 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"4fdfb45900343e3ed47649f8c5003ace7874a541a7afb659d71fc3cb8c8dcb9a"} Nov 03 03:00:32.419095 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.419053 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"42bdc542b917179f75a70d2b218df5650b3a93e2195525a26b9d092babac7e14"} Nov 03 03:00:32.419095 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.419080 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"1f0b0bba53f8e31a7f64f8c3fa2e3eae36cc5e51671fa45b3281accef119c5ff"} Nov 03 03:00:32.419209 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:32.419108 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"0b062f92981f9a86f7a827a1826e438e7ce0bd1c8844c2af418951102b89c0f7"} Nov 03 03:00:34.179723 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.179626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:34.179723 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.179697 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:34.180156 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.179789 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:34.180156 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.179807 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:34.180156 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.179858 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret podName:68969cec-a620-4bd5-bfd1-cd203d0822b8 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:50.179844626 +0000 UTC m=+34.349644664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret") pod "global-pull-secret-syncer-j76qm" (UID: "68969cec-a620-4bd5-bfd1-cd203d0822b8") : object "kube-system"/"original-pull-secret" not registered Nov 03 03:00:34.180156 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.179873 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:50.179866412 +0000 UTC m=+34.349666450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 03:00:34.280705 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.280672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:34.280858 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.280822 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 03:00:34.280858 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.280839 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 03:00:34.280858 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.280848 2563 projected.go:196] Error preparing data for projected volume kube-api-access-r4k7s for pod openshift-network-diagnostics/network-check-target-ldf8b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:34.280957 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.280897 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s podName:4b8aed8e-c303-437e-9e6e-37d9334a1541 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:50.280884446 +0000 UTC m=+34.450684484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4k7s" (UniqueName: "kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s") pod "network-check-target-ldf8b" (UID: "4b8aed8e-c303-437e-9e6e-37d9334a1541") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 03:00:34.316994 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.316963 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:34.317137 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.317027 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:34.317137 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.317029 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:34.317222 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.317124 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:34.317257 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.317224 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:34.317310 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:34.317296 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:34.424268 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.424242 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qg9cd" event={"ID":"724f0110-6fa5-473c-a88b-c514952a50a3","Type":"ContainerStarted","Data":"d2d4ee2aeda1840643de03e5bb513c66acd1c2724a86ba4c1da6d01c69ea0eec"} Nov 03 03:00:34.428377 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.428349 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"5c5a3e085d4bdef288aa49dfe0ff04518049028bfd68f758f2f0723460294f25"} Nov 03 03:00:34.541226 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:34.541121 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qg9cd" podStartSLOduration=3.393999184 podStartE2EDuration="18.541104514s" totalImagesPullingTime="15.14710533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:34.540854087 +0000 UTC m=+18.710654139" watchObservedRunningTime="2026-11-03 03:00:34.541104514 +0000 UTC m=+18.710904576" Nov 03 03:00:36.317325 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:36.317286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:36.317845 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:36.317412 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:36.317845 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:36.317436 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:36.317845 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:36.317525 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:36.317845 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:36.317585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:36.317845 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:36.317710 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:37.247572 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:37.247283 2563 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:37.247993 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:37.247956 2563 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:37.438415 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:37.438380 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" event={"ID":"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af","Type":"ContainerStarted","Data":"55bd637cc47c80492b2d5563d4fa3bdf14dc9a60335ad1c15736ebc77772cd53"} Nov 03 03:00:37.438869 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:37.438622 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:37.439031 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:37.439017 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:00:37.621820 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:37.621758 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" podStartSLOduration=8.343528585 podStartE2EDuration="21.621738166s" totalImagesPullingTime="13.278209581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:37.621203079 +0000 UTC m=+21.791003139" watchObservedRunningTime="2026-11-03 03:00:37.621738166 +0000 UTC m=+21.791538227" Nov 03 03:00:38.317131 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.317094 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:38.317302 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.317158 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:38.317302 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.317171 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:38.317302 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:38.317268 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:38.317483 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:38.317311 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:38.317483 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:38.317343 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:38.442573 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.442543 2563 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="2b47f5b7bd7e58783d6dbc9a954d10edf2d1a072e9395571db372c9bb1bb36ae" exitCode=0 Nov 03 03:00:38.443117 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.442618 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerDied","Data":"2b47f5b7bd7e58783d6dbc9a954d10edf2d1a072e9395571db372c9bb1bb36ae"} Nov 03 03:00:38.444151 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.444123 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" event={"ID":"19f60386-016f-4213-ad61-462deca38b28","Type":"ContainerStarted","Data":"3eefc7ab6ad6a20f25c93733db14a6beef519c3b3de19d74491550ffe7935d8e"} Nov 03 03:00:38.444845 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.444826 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:38.444936 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.444888 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:38.444936 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.444933 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:38.460854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.460822 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:38.460970 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:38.460923 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:00:39.176838 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:39.176797 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j76qm"] Nov 03 03:00:39.177102 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:39.176913 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:39.177102 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:39.177003 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:39.192688 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:39.192656 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xw724"] Nov 03 03:00:39.192842 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:39.192775 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:39.192913 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:39.192891 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:39.194027 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:39.193999 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ldf8b"] Nov 03 03:00:39.194154 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:39.194088 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:39.194209 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:39.194173 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:40.316835 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:40.316801 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:40.317298 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:40.316849 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:40.317298 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:40.316965 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:40.317298 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:40.317079 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:40.451046 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:40.451012 2563 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="c782f0535de8a568c100c6583b5a460eba8a59d966599fc3c4fd6a510bbf09e7" exitCode=0 Nov 03 03:00:40.451207 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:40.451080 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerDied","Data":"c782f0535de8a568c100c6583b5a460eba8a59d966599fc3c4fd6a510bbf09e7"} Nov 03 03:00:41.317006 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:41.316974 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:41.317485 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:41.317070 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:41.813272 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:41.813124 2563 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 03:00:42.316388 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.316280 2563 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T03:00:41.813268543Z","UUID":"4e4ce3ed-745a-4e38-b03b-9fb51cda8c25","Handler":null,"Name":"","Endpoint":""} Nov 03 03:00:42.316613 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.316508 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:42.316613 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.316516 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:42.316719 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:42.316612 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:42.316789 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:42.316767 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:42.317796 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.317777 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 03:00:42.317796 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.317800 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 03:00:42.461143 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.461108 2563 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="ea10e11ee1f9a4a429cf6cb5a7a6c4c80977e674e9676913f3def78022aac3d8" exitCode=0 Nov 03 03:00:42.461307 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.461186 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerDied","Data":"ea10e11ee1f9a4a429cf6cb5a7a6c4c80977e674e9676913f3def78022aac3d8"} Nov 03 03:00:42.463133 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:42.463105 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" event={"ID":"19f60386-016f-4213-ad61-462deca38b28","Type":"ContainerStarted","Data":"dd7bec1755f8b1378fd0c082429395503f9d2cab310e65a69b5c22b24036c522"} Nov 03 03:00:43.317314 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:43.317284 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:43.317491 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:43.317390 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:44.317324 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:44.317285 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:44.317890 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:44.317334 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:44.317890 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:44.317419 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j76qm" podUID="68969cec-a620-4bd5-bfd1-cd203d0822b8" Nov 03 03:00:44.317890 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:44.317548 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xw724" podUID="1714a90e-d9bd-4a76-bc61-295f6978bf17" Nov 03 03:00:45.317264 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.317231 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:45.317431 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.317363 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ldf8b" podUID="4b8aed8e-c303-437e-9e6e-37d9334a1541" Nov 03 03:00:45.473173 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.473077 2563 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="de3e021eb64ee71d17ddf8dde8e12d8d6ae68cf3b974ce4d928a2a0f7c557df0" exitCode=0 Nov 03 03:00:45.473173 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.473148 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerDied","Data":"de3e021eb64ee71d17ddf8dde8e12d8d6ae68cf3b974ce4d928a2a0f7c557df0"} Nov 03 03:00:45.606952 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.606930 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeReady" Nov 03 03:00:45.607064 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.607054 2563 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 03:00:45.701523 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.701496 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d8k4b"] Nov 03 03:00:45.705440 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.705420 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nz8pk"] Nov 03 03:00:45.705564 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.705549 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" Nov 03 03:00:45.708068 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.708040 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5r574"] Nov 03 03:00:45.708349 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.708327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" Nov 03 03:00:45.711021 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.711003 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq"] Nov 03 03:00:45.711130 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.711101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5r574" Nov 03 03:00:45.713627 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.713609 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c5c7c7bf7-x5tt5"] Nov 03 03:00:45.713729 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.713715 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.716195 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.716178 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-f869c"] Nov 03 03:00:45.716301 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.716287 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.718861 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.718846 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2hdcl"] Nov 03 03:00:45.718961 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.718948 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.721297 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.721283 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4"] Nov 03 03:00:45.721398 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.721384 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.726055 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.724585 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr"] Nov 03 03:00:45.726055 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.724690 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.727579 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.727565 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd"] Nov 03 03:00:45.727689 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.727673 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.730139 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.730120 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675ff945d6-2qfxd"] Nov 03 03:00:45.730236 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.730189 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" Nov 03 03:00:45.732791 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.732773 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bhczt"] Nov 03 03:00:45.732881 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.732873 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.735367 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.735348 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bf7f94cf9-nvxkf"] Nov 03 03:00:45.735470 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.735454 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:45.738995 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.738976 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f"] Nov 03 03:00:45.739151 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.739103 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.741873 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.741845 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm"] Nov 03 03:00:45.742049 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.741920 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.744776 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.744757 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lctfx"] Nov 03 03:00:45.744881 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.744867 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" Nov 03 03:00:45.748153 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.748131 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dwkvw"] Nov 03 03:00:45.748265 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.748253 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:45.751383 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.751366 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.767009 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.763284 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.767009 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.763372 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.767009 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.764870 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d8k4b"] Nov 03 03:00:45.767009 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.766960 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nz8pk"] Nov 03 03:00:45.769313 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.769288 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 03:00:45.780564 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.780546 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 03:00:45.780721 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.780704 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 03:00:45.780862 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.780846 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 03:00:45.781267 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.781247 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 03:00:45.782593 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.782576 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 03:00:45.808109 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.808084 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c5c7c7bf7-x5tt5"] Nov 03 03:00:45.810290 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.810257 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq"] Nov 03 03:00:45.810731 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.810714 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5r574"] Nov 03 03:00:45.810885 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.810865 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2hdcl"] Nov 03 03:00:45.839886 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.839865 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4"] Nov 03 03:00:45.853548 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.853520 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd"] Nov 03 03:00:45.853827 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.853788 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr"] Nov 03 03:00:45.854674 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.854646 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f"] Nov 03 03:00:45.856184 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.856164 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-f869c"] Nov 03 03:00:45.875156 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.875137 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 03:00:45.875335 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.875311 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.875401 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.875386 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 03:00:45.875715 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.875703 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.875925 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.875912 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-default-certificate\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.875956 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.875935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-service-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.876001 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.875990 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmg4d\" (UniqueName: \"kubernetes.io/projected/c02e7385-46cc-46a6-822f-f47fe2819a3f-kube-api-access-gmg4d\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.876036 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876009 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-config\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.876036 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.876109 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.876109 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876076 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.876109 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.876234 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876127 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.876234 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876154 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.876234 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.876360 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crsn5\" (UniqueName: \"kubernetes.io/projected/83b404fd-1b9b-4088-ba59-d9997f5a1558-kube-api-access-crsn5\") pod \"kas-connection-checker-685895c7c9-nz8pk\" (UID: \"83b404fd-1b9b-4088-ba59-d9997f5a1558\") " pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" Nov 03 03:00:45.876360 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtspf\" (UniqueName: \"kubernetes.io/projected/8fa09e76-0e81-4449-a649-1b5705b8d5bd-kube-api-access-qtspf\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.876472 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876431 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:45.876531 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876489 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.876531 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876499 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.876614 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876534 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5w4\" (UniqueName: \"kubernetes.io/projected/50ae62be-5346-4aff-b350-305a9a5e1a6d-kube-api-access-2g5w4\") pod \"kas-connection-checker-685895c7c9-5r574\" (UID: \"50ae62be-5346-4aff-b350-305a9a5e1a6d\") " pod="kube-system/kas-connection-checker-685895c7c9-5r574" Nov 03 03:00:45.876614 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876572 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.876678 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876645 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.876678 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-snapshots\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.876743 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876712 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.876793 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876767 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.876838 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876819 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrcw\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.876908 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.876951 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktp62\" (UniqueName: \"kubernetes.io/projected/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-kube-api-access-ktp62\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.876988 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.876988 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.876982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.877059 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877000 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/297e0916-b3bb-4002-810c-390a724cd782-serving-cert\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.877059 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877025 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 03:00:45.877059 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.877162 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-tmp-dir\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.877162 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877072 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.877162 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877146 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-tmp\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.877264 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877212 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.877302 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gr2\" (UniqueName: \"kubernetes.io/projected/f58a099e-243b-4581-bebc-8ceaf23509c4-kube-api-access-c4gr2\") pod \"network-check-source-6f78d59f99-tjzsm\" (UID: \"f58a099e-243b-4581-bebc-8ceaf23509c4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" Nov 03 03:00:45.877345 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-cert\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:45.877401 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lhw\" (UniqueName: \"kubernetes.io/projected/0e4b8195-7644-4736-af66-d32d0d48551e-kube-api-access-42lhw\") pod \"kas-connection-checker-685895c7c9-d8k4b\" (UID: \"0e4b8195-7644-4736-af66-d32d0d48551e\") " pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" Nov 03 03:00:45.877433 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877421 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.877494 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877467 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.877539 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877496 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-trusted-ca\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.877539 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877526 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59642d89-3475-44f2-8ddd-c95da8e30f44-serving-cert\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.877892 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877561 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.877969 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877915 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5fl\" (UniqueName: \"kubernetes.io/projected/297e0916-b3bb-4002-810c-390a724cd782-kube-api-access-lf5fl\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.877969 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877950 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8lc\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.878071 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.877978 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.878071 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878035 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 03:00:45.878071 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878036 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gl9\" (UniqueName: \"kubernetes.io/projected/fef674c6-31e4-4f47-b723-19bc2b38502c-kube-api-access-79gl9\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.878071 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878058 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.878250 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878129 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm"] Nov 03 03:00:45.878250 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878152 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5tr\" (UniqueName: \"kubernetes.io/projected/2469c20c-30a7-41bb-b726-d62d9324f693-kube-api-access-gw5tr\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.878250 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878191 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.878250 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878221 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fef674c6-31e4-4f47-b723-19bc2b38502c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.878411 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878339 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.878411 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878368 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4j48\" (UniqueName: \"kubernetes.io/projected/59642d89-3475-44f2-8ddd-c95da8e30f44-kube-api-access-h4j48\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.878411 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878395 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:45.878411 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878391 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 03:00:45.878629 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878435 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-stats-auth\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.878629 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878437 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.878629 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878513 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.878629 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tl5c\" (UniqueName: \"kubernetes.io/projected/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-kube-api-access-5tl5c\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.878629 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878616 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa09e76-0e81-4449-a649-1b5705b8d5bd-serving-cert\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.878835 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878651 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-config-volume\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.878835 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-777xw\" (UniqueName: \"kubernetes.io/projected/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-kube-api-access-777xw\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:45.878835 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878703 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f892q\" (UniqueName: \"kubernetes.io/projected/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2-kube-api-access-f892q\") pod \"volume-data-source-validator-58c9f5f46d-rgsmd\" (UID: \"09f7a7ee-de2c-4c6d-8ed8-39d5405848e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" Nov 03 03:00:45.878835 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878741 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.878835 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.878818 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59642d89-3475-44f2-8ddd-c95da8e30f44-config\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.901430 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.901390 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 03:00:45.901430 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.901415 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.905578 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.905556 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 03:00:45.925037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.925007 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.926064 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.926043 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.927241 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.927227 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.927507 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.927492 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.927713 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.927701 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.928076 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.928065 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.928314 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.928303 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.938341 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.938317 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 03:00:45.950187 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950159 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 03:00:45.950315 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950298 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.950618 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950592 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 03:00:45.950618 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950598 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 03:00:45.950782 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950679 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 03:00:45.950782 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950687 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 03:00:45.950782 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950689 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 03:00:45.950782 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950698 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 03:00:45.950782 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950750 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 03:00:45.950969 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950891 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.950969 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950916 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 03:00:45.950969 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950945 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 03:00:45.950969 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950967 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 03:00:45.951129 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950975 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 03:00:45.951129 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950987 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 03:00:45.951129 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.950972 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 03:00:45.951129 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.951009 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 03:00:45.951129 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.951054 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zgcvz\"" Nov 03 03:00:45.951288 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.951160 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pw7w9\"" Nov 03 03:00:45.951288 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.951210 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tbkxl\"" Nov 03 03:00:45.951349 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.951287 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snr2z\"" Nov 03 03:00:45.952111 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.952097 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bhczt"] Nov 03 03:00:45.955942 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.955901 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675ff945d6-2qfxd"] Nov 03 03:00:45.956467 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.956426 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bf7f94cf9-nvxkf"] Nov 03 03:00:45.956756 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.956738 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lctfx"] Nov 03 03:00:45.957388 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.957369 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dwkvw"] Nov 03 03:00:45.979428 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.979428 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fef674c6-31e4-4f47-b723-19bc2b38502c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.979547 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979438 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.979547 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979477 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j48\" (UniqueName: \"kubernetes.io/projected/59642d89-3475-44f2-8ddd-c95da8e30f44-kube-api-access-h4j48\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.979547 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979536 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:45.979684 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-stats-auth\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.979684 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.979773 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979673 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tl5c\" (UniqueName: \"kubernetes.io/projected/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-kube-api-access-5tl5c\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.979773 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979733 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa09e76-0e81-4449-a649-1b5705b8d5bd-serving-cert\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.979864 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979769 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-config-volume\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.979864 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979811 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-777xw\" (UniqueName: \"kubernetes.io/projected/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-kube-api-access-777xw\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:45.979864 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979840 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f892q\" (UniqueName: \"kubernetes.io/projected/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2-kube-api-access-f892q\") pod \"volume-data-source-validator-58c9f5f46d-rgsmd\" (UID: \"09f7a7ee-de2c-4c6d-8ed8-39d5405848e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" Nov 03 03:00:45.979983 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.979983 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59642d89-3475-44f2-8ddd-c95da8e30f44-config\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.979983 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-default-certificate\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.980133 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-service-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.980133 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.979998 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.980133 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmg4d\" (UniqueName: \"kubernetes.io/projected/c02e7385-46cc-46a6-822f-f47fe2819a3f-kube-api-access-gmg4d\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.980133 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-config\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.980133 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980093 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.980133 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.980487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980155 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.980487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.980487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980225 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.980487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980231 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.980487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980353 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:45.980487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980511 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsn5\" (UniqueName: \"kubernetes.io/projected/83b404fd-1b9b-4088-ba59-d9997f5a1558-kube-api-access-crsn5\") pod \"kas-connection-checker-685895c7c9-nz8pk\" (UID: \"83b404fd-1b9b-4088-ba59-d9997f5a1558\") " pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980540 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtspf\" (UniqueName: \"kubernetes.io/projected/8fa09e76-0e81-4449-a649-1b5705b8d5bd-kube-api-access-qtspf\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5w4\" (UniqueName: \"kubernetes.io/projected/50ae62be-5346-4aff-b350-305a9a5e1a6d-kube-api-access-2g5w4\") pod \"kas-connection-checker-685895c7c9-5r574\" (UID: \"50ae62be-5346-4aff-b350-305a9a5e1a6d\") " pod="kube-system/kas-connection-checker-685895c7c9-5r574" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980648 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980678 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-snapshots\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.980766 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrcw\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktp62\" (UniqueName: \"kubernetes.io/projected/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-kube-api-access-ktp62\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980878 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980906 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/297e0916-b3bb-4002-810c-390a724cd782-serving-cert\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980936 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-tmp-dir\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980978 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.980989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-tmp\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981039 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gr2\" (UniqueName: \"kubernetes.io/projected/f58a099e-243b-4581-bebc-8ceaf23509c4-kube-api-access-c4gr2\") pod \"network-check-source-6f78d59f99-tjzsm\" (UID: \"f58a099e-243b-4581-bebc-8ceaf23509c4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981104 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-cert\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981137 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lhw\" (UniqueName: \"kubernetes.io/projected/0e4b8195-7644-4736-af66-d32d0d48551e-kube-api-access-42lhw\") pod \"kas-connection-checker-685895c7c9-d8k4b\" (UID: \"0e4b8195-7644-4736-af66-d32d0d48551e\") " pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981167 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.981208 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-trusted-ca\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59642d89-3475-44f2-8ddd-c95da8e30f44-serving-cert\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981272 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5fl\" (UniqueName: \"kubernetes.io/projected/297e0916-b3bb-4002-810c-390a724cd782-kube-api-access-lf5fl\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981298 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-tmp\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981319 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8lc\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981343 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gl9\" (UniqueName: \"kubernetes.io/projected/fef674c6-31e4-4f47-b723-19bc2b38502c-kube-api-access-79gl9\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5tr\" (UniqueName: \"kubernetes.io/projected/2469c20c-30a7-41bb-b726-d62d9324f693-kube-api-access-gw5tr\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:45.981912 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981669 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59642d89-3475-44f2-8ddd-c95da8e30f44-config\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981938 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-config\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.982119 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fef674c6-31e4-4f47-b723-19bc2b38502c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982250 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.982289 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982486 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.482380144 +0000 UTC m=+30.652180184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982507 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.482496559 +0000 UTC m=+30.652296596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982519 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 03:00:45.982582 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982568 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls podName:2469c20c-30a7-41bb-b726-d62d9324f693 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.482551134 +0000 UTC m=+30.652351172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t2pr" (UID: "2469c20c-30a7-41bb-b726-d62d9324f693") : secret "samples-operator-tls" not found Nov 03 03:00:45.982949 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982734 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:45.982949 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982781 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls podName:fef674c6-31e4-4f47-b723-19bc2b38502c nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.482766892 +0000 UTC m=+30.652566953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-p7hdq" (UID: "fef674c6-31e4-4f47-b723-19bc2b38502c") : secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:45.982949 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.982780 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.982949 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982798 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:00:45.982949 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.982851 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.48283361 +0000 UTC m=+30.652633674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:00:45.983190 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.983041 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:45.983190 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.983054 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675ff945d6-2qfxd: secret "image-registry-tls" not found Nov 03 03:00:45.983190 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.983105 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls podName:9db36b8c-a3dd-4bb0-8710-1d0c09e981cd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.483091284 +0000 UTC m=+30.652891323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls") pod "image-registry-675ff945d6-2qfxd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd") : secret "image-registry-tls" not found Nov 03 03:00:45.983190 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.983127 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-snapshots\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.983758 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.983406 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.984174 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.984149 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-config-volume\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.984252 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.984209 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-tmp-dir\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:45.984252 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.981297 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-service-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.984359 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.984334 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 03:00:45.984413 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.984385 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls podName:f2b7b1fa-14b8-47ee-a3ce-dd236209b96b nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.48437224 +0000 UTC m=+30.654172278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls") pod "dns-default-dwkvw" (UID: "f2b7b1fa-14b8-47ee-a3ce-dd236209b96b") : secret "dns-default-metrics-tls" not found Nov 03 03:00:45.985045 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.985021 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.985045 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.985031 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.985049 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7f94cf9-nvxkf: secret "image-registry-tls" not found Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:45.985204 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls podName:72608a5a-a7a0-43e7-a9c0-bc681d988262 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:46.485187098 +0000 UTC m=+30.654987157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls") pod "image-registry-5bf7f94cf9-nvxkf" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262") : secret "image-registry-tls" not found Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.985467 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-stats-auth\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.985463 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-default-certificate\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.985606 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa09e76-0e81-4449-a649-1b5705b8d5bd-serving-cert\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.985717 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.985728 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.985838 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:45.986391 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.986215 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-trusted-ca\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.986939 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.986915 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59642d89-3475-44f2-8ddd-c95da8e30f44-serving-cert\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:45.987551 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.987529 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.987648 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.987607 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/297e0916-b3bb-4002-810c-390a724cd782-serving-cert\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:45.987744 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.987728 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:45.987853 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.987836 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:45.988585 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:45.988570 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-cert\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:46.044497 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.044470 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4j48\" (UniqueName: \"kubernetes.io/projected/59642d89-3475-44f2-8ddd-c95da8e30f44-kube-api-access-h4j48\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:46.044789 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.044761 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:46.044997 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.044976 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tl5c\" (UniqueName: \"kubernetes.io/projected/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-kube-api-access-5tl5c\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:46.045355 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.045332 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrcw\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:46.045702 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.045683 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-777xw\" (UniqueName: \"kubernetes.io/projected/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-kube-api-access-777xw\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:46.045774 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.045748 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5w4\" (UniqueName: \"kubernetes.io/projected/50ae62be-5346-4aff-b350-305a9a5e1a6d-kube-api-access-2g5w4\") pod \"kas-connection-checker-685895c7c9-5r574\" (UID: \"50ae62be-5346-4aff-b350-305a9a5e1a6d\") " pod="kube-system/kas-connection-checker-685895c7c9-5r574" Nov 03 03:00:46.046413 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.046394 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsn5\" (UniqueName: \"kubernetes.io/projected/83b404fd-1b9b-4088-ba59-d9997f5a1558-kube-api-access-crsn5\") pod \"kas-connection-checker-685895c7c9-nz8pk\" (UID: \"83b404fd-1b9b-4088-ba59-d9997f5a1558\") " pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" Nov 03 03:00:46.058915 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.058890 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:46.060089 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.060065 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktp62\" (UniqueName: \"kubernetes.io/projected/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-kube-api-access-ktp62\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:46.061524 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.061491 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmg4d\" (UniqueName: \"kubernetes.io/projected/c02e7385-46cc-46a6-822f-f47fe2819a3f-kube-api-access-gmg4d\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:46.061805 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.061781 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8lc\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:46.061888 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.061849 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gl9\" (UniqueName: \"kubernetes.io/projected/fef674c6-31e4-4f47-b723-19bc2b38502c-kube-api-access-79gl9\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:46.062132 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.062114 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f892q\" (UniqueName: \"kubernetes.io/projected/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2-kube-api-access-f892q\") pod \"volume-data-source-validator-58c9f5f46d-rgsmd\" (UID: \"09f7a7ee-de2c-4c6d-8ed8-39d5405848e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" Nov 03 03:00:46.062436 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.062422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gr2\" (UniqueName: \"kubernetes.io/projected/f58a099e-243b-4581-bebc-8ceaf23509c4-kube-api-access-c4gr2\") pod \"network-check-source-6f78d59f99-tjzsm\" (UID: \"f58a099e-243b-4581-bebc-8ceaf23509c4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" Nov 03 03:00:46.062956 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.062934 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lhw\" (UniqueName: \"kubernetes.io/projected/0e4b8195-7644-4736-af66-d32d0d48551e-kube-api-access-42lhw\") pod \"kas-connection-checker-685895c7c9-d8k4b\" (UID: \"0e4b8195-7644-4736-af66-d32d0d48551e\") " pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" Nov 03 03:00:46.064610 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.064590 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5tr\" (UniqueName: \"kubernetes.io/projected/2469c20c-30a7-41bb-b726-d62d9324f693-kube-api-access-gw5tr\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:46.065189 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.065170 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtspf\" (UniqueName: \"kubernetes.io/projected/8fa09e76-0e81-4449-a649-1b5705b8d5bd-kube-api-access-qtspf\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:46.065291 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.065276 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5fl\" (UniqueName: \"kubernetes.io/projected/297e0916-b3bb-4002-810c-390a724cd782-kube-api-access-lf5fl\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:46.066088 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.066074 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:46.076403 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.076385 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:00:46.081879 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.081862 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:00:46.092947 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.092916 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" Nov 03 03:00:46.123621 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.123584 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:00:46.126725 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.126704 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" Nov 03 03:00:46.132881 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.132349 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:00:46.316622 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.316597 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:46.316824 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.316811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:46.331710 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.331692 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" Nov 03 03:00:46.336492 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.336470 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" Nov 03 03:00:46.343850 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.343829 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5r574" Nov 03 03:00:46.379860 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.379834 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 03:00:46.380000 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.379941 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ckmp9\"" Nov 03 03:00:46.380062 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.380005 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\"" Nov 03 03:00:46.380143 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.380130 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 03:00:46.471465 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.471399 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2hdcl"] Nov 03 03:00:46.471618 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.471492 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4"] Nov 03 03:00:46.495138 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.490377 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-f869c"] Nov 03 03:00:46.495479 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:46.495479 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:46.495479 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:46.495635 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:46.495635 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:46.495635 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:46.495635 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:46.495792 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.495662 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:46.495880 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495840 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:00:46.495929 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495896 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:46.495929 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495899 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 03:00:46.495929 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495913 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7f94cf9-nvxkf: secret "image-registry-tls" not found Nov 03 03:00:46.496043 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495909 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.495889855 +0000 UTC m=+31.665689909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:00:46.496043 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495972 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls podName:2469c20c-30a7-41bb-b726-d62d9324f693 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.495956387 +0000 UTC m=+31.665756428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t2pr" (UID: "2469c20c-30a7-41bb-b726-d62d9324f693") : secret "samples-operator-tls" not found Nov 03 03:00:46.496043 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495985 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls podName:72608a5a-a7a0-43e7-a9c0-bc681d988262 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.495977685 +0000 UTC m=+31.665777725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls") pod "image-registry-5bf7f94cf9-nvxkf" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262") : secret "image-registry-tls" not found Nov 03 03:00:46.496043 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495841 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:46.496043 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495998 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675ff945d6-2qfxd: secret "image-registry-tls" not found Nov 03 03:00:46.496043 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.496027 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls podName:9db36b8c-a3dd-4bb0-8710-1d0c09e981cd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.496017476 +0000 UTC m=+31.665817535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls") pod "image-registry-675ff945d6-2qfxd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd") : secret "image-registry-tls" not found Nov 03 03:00:46.496309 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.496153 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.496136721 +0000 UTC m=+31.665936773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:00:46.496309 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.496218 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:00:46.496309 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.495840 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 03:00:46.496309 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.496255 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.49624378 +0000 UTC m=+31.666043823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:00:46.496309 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.496293 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls podName:f2b7b1fa-14b8-47ee-a3ce-dd236209b96b nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.496279186 +0000 UTC m=+31.666079237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls") pod "dns-default-dwkvw" (UID: "f2b7b1fa-14b8-47ee-a3ce-dd236209b96b") : secret "dns-default-metrics-tls" not found Nov 03 03:00:46.496309 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.496308 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:46.496589 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:46.496343 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls podName:fef674c6-31e4-4f47-b723-19bc2b38502c nodeName:}" failed. No retries permitted until 2026-11-03 03:00:47.496331215 +0000 UTC m=+31.666131253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-p7hdq" (UID: "fef674c6-31e4-4f47-b723-19bc2b38502c") : secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:46.502046 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.502021 2563 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="ade6fc237b1fc4f1f6b2f031aff989babf7dab808a6185d6c6adee6c741f26d3" exitCode=0 Nov 03 03:00:46.502138 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.502098 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerDied","Data":"ade6fc237b1fc4f1f6b2f031aff989babf7dab808a6185d6c6adee6c741f26d3"} Nov 03 03:00:46.504567 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.504544 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" event={"ID":"19f60386-016f-4213-ad61-462deca38b28","Type":"ContainerStarted","Data":"f0e8a198f4fa05d938a78c15dfe004eb7ea56301505f1af75cc4dff9c4f25b31"} Nov 03 03:00:46.505882 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.505832 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2hdcl" event={"ID":"8fa09e76-0e81-4449-a649-1b5705b8d5bd","Type":"ContainerStarted","Data":"a579d256c87d8443d30c12e6b1124b5846be4b75db4facfd54e456d212857253"} Nov 03 03:00:46.507270 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.507249 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd"] Nov 03 03:00:46.511388 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:46.510726 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f7a7ee_de2c_4c6d_8ed8_39d5405848e2.slice/crio-8649e252d12ca03cf941fca6ac569943d9701612bebb3f3f385ecd5e64909b5b WatchSource:0}: Error finding container 8649e252d12ca03cf941fca6ac569943d9701612bebb3f3f385ecd5e64909b5b: Status 404 returned error can't find the container with id 8649e252d12ca03cf941fca6ac569943d9701612bebb3f3f385ecd5e64909b5b Nov 03 03:00:46.570199 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.570171 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm"] Nov 03 03:00:46.573392 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.573372 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f"] Nov 03 03:00:46.687410 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.687373 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lctfx"] Nov 03 03:00:46.690498 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:46.690469 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc36c84_f2e0_4f30_b74e_73d169c91ca8.slice/crio-05f9c340ef08fe05ccae8d80c2f769fa5d71307f6bc1aa44aac440a79296e34c WatchSource:0}: Error finding container 05f9c340ef08fe05ccae8d80c2f769fa5d71307f6bc1aa44aac440a79296e34c: Status 404 returned error can't find the container with id 05f9c340ef08fe05ccae8d80c2f769fa5d71307f6bc1aa44aac440a79296e34c Nov 03 03:00:46.704886 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.704541 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d8k4b"] Nov 03 03:00:46.708098 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:46.708070 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4b8195_7644_4736_af66_d32d0d48551e.slice/crio-9194453833f4e13774ca808f943858755243ffde3a9fa79e90e2aafa829a0fef WatchSource:0}: Error finding container 9194453833f4e13774ca808f943858755243ffde3a9fa79e90e2aafa829a0fef: Status 404 returned error can't find the container with id 9194453833f4e13774ca808f943858755243ffde3a9fa79e90e2aafa829a0fef Nov 03 03:00:46.713819 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.713793 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nz8pk"] Nov 03 03:00:46.716506 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:46.716485 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b404fd_1b9b_4088_ba59_d9997f5a1558.slice/crio-8b9ba714f837281310eecb088372c941689d8434e845664f97fca911d45a5aa0 WatchSource:0}: Error finding container 8b9ba714f837281310eecb088372c941689d8434e845664f97fca911d45a5aa0: Status 404 returned error can't find the container with id 8b9ba714f837281310eecb088372c941689d8434e845664f97fca911d45a5aa0 Nov 03 03:00:46.738610 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:46.738581 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5r574"] Nov 03 03:00:46.742211 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:46.742179 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ae62be_5346_4aff_b350_305a9a5e1a6d.slice/crio-1e55be5b8a42a19bc19296ec69498e669824aa2de0bcf8864043e5128c761431 WatchSource:0}: Error finding container 1e55be5b8a42a19bc19296ec69498e669824aa2de0bcf8864043e5128c761431: Status 404 returned error can't find the container with id 1e55be5b8a42a19bc19296ec69498e669824aa2de0bcf8864043e5128c761431 Nov 03 03:00:47.316907 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.316875 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:47.370001 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.369970 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\"" Nov 03 03:00:47.504848 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.504806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:47.505003 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.504863 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:47.505003 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.504958 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:00:47.505003 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.504963 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:00:47.505127 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505014 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505000099 +0000 UTC m=+33.674800153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:00:47.505127 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505029 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505022274 +0000 UTC m=+33.674822312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:00:47.505127 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.505045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:47.505127 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.505098 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:47.505127 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.505121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:47.505277 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.505140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:47.505277 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.505160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:47.505277 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.505188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:47.505376 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505299 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:47.505376 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505311 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7f94cf9-nvxkf: secret "image-registry-tls" not found Nov 03 03:00:47.505376 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505339 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls podName:72608a5a-a7a0-43e7-a9c0-bc681d988262 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505330275 +0000 UTC m=+33.675130327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls") pod "image-registry-5bf7f94cf9-nvxkf" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262") : secret "image-registry-tls" not found Nov 03 03:00:47.505492 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505378 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:47.505492 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505407 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls podName:fef674c6-31e4-4f47-b723-19bc2b38502c nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505392319 +0000 UTC m=+33.675192371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-p7hdq" (UID: "fef674c6-31e4-4f47-b723-19bc2b38502c") : secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:47.505492 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505485 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505477688 +0000 UTC m=+33.675277739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:00:47.505630 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505534 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 03:00:47.505630 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505562 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls podName:2469c20c-30a7-41bb-b726-d62d9324f693 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505554452 +0000 UTC m=+33.675354503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t2pr" (UID: "2469c20c-30a7-41bb-b726-d62d9324f693") : secret "samples-operator-tls" not found Nov 03 03:00:47.505630 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505600 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:47.505630 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505620 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675ff945d6-2qfxd: secret "image-registry-tls" not found Nov 03 03:00:47.505630 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505620 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 03:00:47.505815 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505639 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls podName:9db36b8c-a3dd-4bb0-8710-1d0c09e981cd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505633348 +0000 UTC m=+33.675433400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls") pod "image-registry-675ff945d6-2qfxd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd") : secret "image-registry-tls" not found Nov 03 03:00:47.505815 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:47.505672 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls podName:f2b7b1fa-14b8-47ee-a3ce-dd236209b96b nodeName:}" failed. No retries permitted until 2026-11-03 03:00:49.505654893 +0000 UTC m=+33.675454932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls") pod "dns-default-dwkvw" (UID: "f2b7b1fa-14b8-47ee-a3ce-dd236209b96b") : secret "dns-default-metrics-tls" not found Nov 03 03:00:47.512851 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.512797 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" event={"ID":"57dbddca-a044-4478-8cfe-9867494e4291","Type":"ContainerStarted","Data":"d05ca772c6a5ce19033dbeebe38ba024e6a78b7051b3cd9b9c3dd4c45e86fd3c"} Nov 03 03:00:47.514111 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.514080 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" event={"ID":"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a","Type":"ContainerStarted","Data":"9613fd2e56776a7bfcf8d988b7c71ff9de0a6867f2f7f1befdbd2f9d1357a0f1"} Nov 03 03:00:47.515249 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.515221 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" event={"ID":"59642d89-3475-44f2-8ddd-c95da8e30f44","Type":"ContainerStarted","Data":"78c9f862d266c6d3d4de307c7332aeca1b614cb3782755bb2b2530d93924756a"} Nov 03 03:00:47.516595 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.516574 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" event={"ID":"0e4b8195-7644-4736-af66-d32d0d48551e","Type":"ContainerStarted","Data":"6ad413395082abe034d042b084ae0c5aaf1de41fb00d815e6b6a4b87c4e4d126"} Nov 03 03:00:47.516678 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.516612 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" event={"ID":"0e4b8195-7644-4736-af66-d32d0d48551e","Type":"ContainerStarted","Data":"9194453833f4e13774ca808f943858755243ffde3a9fa79e90e2aafa829a0fef"} Nov 03 03:00:47.517877 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.517857 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" event={"ID":"83b404fd-1b9b-4088-ba59-d9997f5a1558","Type":"ContainerStarted","Data":"49551976d9fe43bbc9136e7b630e6fe440197a69ff52f921b0314e725cdc247e"} Nov 03 03:00:47.517959 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.517891 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" event={"ID":"83b404fd-1b9b-4088-ba59-d9997f5a1558","Type":"ContainerStarted","Data":"8b9ba714f837281310eecb088372c941689d8434e845664f97fca911d45a5aa0"} Nov 03 03:00:47.519167 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.519141 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" event={"ID":"f58a099e-243b-4581-bebc-8ceaf23509c4","Type":"ContainerStarted","Data":"c0e86de06d203ddb4d647ed53a14c73f8e4cd96aef670ea2ab644f67588d7920"} Nov 03 03:00:47.520506 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.520485 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" event={"ID":"297e0916-b3bb-4002-810c-390a724cd782","Type":"ContainerStarted","Data":"b06265fc4458ab46cac74ec59b2a8f9963c357920e1335ad783c04dea34319a8"} Nov 03 03:00:47.521822 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.521800 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5r574" event={"ID":"50ae62be-5346-4aff-b350-305a9a5e1a6d","Type":"ContainerStarted","Data":"cbc66b7075c06ec0cb46038062008ab4e936f92036b9034ee487ae352aa701f2"} Nov 03 03:00:47.521926 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.521839 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5r574" event={"ID":"50ae62be-5346-4aff-b350-305a9a5e1a6d","Type":"ContainerStarted","Data":"1e55be5b8a42a19bc19296ec69498e669824aa2de0bcf8864043e5128c761431"} Nov 03 03:00:47.523162 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.523142 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lctfx" event={"ID":"3fc36c84-f2e0-4f30-b74e-73d169c91ca8","Type":"ContainerStarted","Data":"05f9c340ef08fe05ccae8d80c2f769fa5d71307f6bc1aa44aac440a79296e34c"} Nov 03 03:00:47.524826 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.524750 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" event={"ID":"09f7a7ee-de2c-4c6d-8ed8-39d5405848e2","Type":"ContainerStarted","Data":"8649e252d12ca03cf941fca6ac569943d9701612bebb3f3f385ecd5e64909b5b"} Nov 03 03:00:47.948026 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:47.946077 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5r574" podStartSLOduration=404.946058529 podStartE2EDuration="6m44.946058529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:47.941610304 +0000 UTC m=+32.111410380" watchObservedRunningTime="2026-11-03 03:00:47.946058529 +0000 UTC m=+32.115858590" Nov 03 03:00:48.062926 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:48.062863 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" podStartSLOduration=5.331555489 podStartE2EDuration="32.062842087s" totalImagesPullingTime="26.731286598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:48.061244873 +0000 UTC m=+32.231044948" watchObservedRunningTime="2026-11-03 03:00:48.062842087 +0000 UTC m=+32.232642150" Nov 03 03:00:48.186331 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:48.186264 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hzvq6" podStartSLOduration=9.350311208 podStartE2EDuration="32.186243648s" totalImagesPullingTime="22.83593244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:48.186002468 +0000 UTC m=+32.355802528" watchObservedRunningTime="2026-11-03 03:00:48.186243648 +0000 UTC m=+32.356043705" Nov 03 03:00:48.307538 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:48.307412 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" podStartSLOduration=405.307393207 podStartE2EDuration="6m45.307393207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:48.306210547 +0000 UTC m=+32.476010627" watchObservedRunningTime="2026-11-03 03:00:48.307393207 +0000 UTC m=+32.477193264" Nov 03 03:00:48.429431 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:48.429373 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" podStartSLOduration=405.429354402 podStartE2EDuration="6m45.429354402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:48.425517694 +0000 UTC m=+32.595317780" watchObservedRunningTime="2026-11-03 03:00:48.429354402 +0000 UTC m=+32.599154462" Nov 03 03:00:49.526484 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526545 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526585 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526657 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.526638194 +0000 UTC m=+37.696438246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526672 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526681 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526689 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.526678097 +0000 UTC m=+37.696478152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526759 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526811 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526836 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526848 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526869 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls podName:fef674c6-31e4-4f47-b723-19bc2b38502c nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.526852707 +0000 UTC m=+37.696652752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-p7hdq" (UID: "fef674c6-31e4-4f47-b723-19bc2b38502c") : secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:49.526987 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526894 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.526883163 +0000 UTC m=+37.696683201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526912 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls podName:2469c20c-30a7-41bb-b726-d62d9324f693 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.526902351 +0000 UTC m=+37.696702405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t2pr" (UID: "2469c20c-30a7-41bb-b726-d62d9324f693") : secret "samples-operator-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526921 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526932 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526934 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675ff945d6-2qfxd: secret "image-registry-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.526948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.526975 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls podName:f2b7b1fa-14b8-47ee-a3ce-dd236209b96b nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.526961074 +0000 UTC m=+37.696761111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls") pod "dns-default-dwkvw" (UID: "f2b7b1fa-14b8-47ee-a3ce-dd236209b96b") : secret "dns-default-metrics-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.527003 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.527006 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls podName:9db36b8c-a3dd-4bb0-8710-1d0c09e981cd nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.526996367 +0000 UTC m=+37.696796409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls") pod "image-registry-675ff945d6-2qfxd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd") : secret "image-registry-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.527012 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7f94cf9-nvxkf: secret "image-registry-tls" not found Nov 03 03:00:49.527518 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:49.527039 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls podName:72608a5a-a7a0-43e7-a9c0-bc681d988262 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.527029361 +0000 UTC m=+37.696829436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls") pod "image-registry-5bf7f94cf9-nvxkf" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262") : secret "image-registry-tls" not found Nov 03 03:00:49.537594 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:49.537563 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" event={"ID":"09f7a7ee-de2c-4c6d-8ed8-39d5405848e2","Type":"ContainerStarted","Data":"73d41e3cb430f50b1e09693523ebae4720b6f8a44485972aadff3a85397e3859"} Nov 03 03:00:50.233186 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.233148 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:50.233186 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.233191 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:00:50.233535 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:50.233308 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 03:00:50.233535 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:50.233373 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:22.23335676 +0000 UTC m=+66.403156814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : secret "metrics-daemon-secret" not found Nov 03 03:00:50.235708 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.235683 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:50.333865 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.333833 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:50.336547 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.336530 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:50.530124 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.530052 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:00:50.546531 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.546500 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/0.log" Nov 03 03:00:50.546694 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.546544 2563 generic.go:356] "Generic (PLEG): container finished" podID="297e0916-b3bb-4002-810c-390a724cd782" containerID="cf4f46f1a3dd96cde804654cda0b893ca6d5c4cbc1714e9546b64b4ec8a46c0f" exitCode=255 Nov 03 03:00:50.546694 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.546643 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" event={"ID":"297e0916-b3bb-4002-810c-390a724cd782","Type":"ContainerDied","Data":"cf4f46f1a3dd96cde804654cda0b893ca6d5c4cbc1714e9546b64b4ec8a46c0f"} Nov 03 03:00:50.546876 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.546858 2563 scope.go:122] "RemoveContainer" containerID="cf4f46f1a3dd96cde804654cda0b893ca6d5c4cbc1714e9546b64b4ec8a46c0f" Nov 03 03:00:50.628316 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.628119 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:50.670836 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.670427 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" podStartSLOduration=375.515633447 podStartE2EDuration="6m17.670403974s" totalImagesPullingTime="2.154770527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:49.661408755 +0000 UTC m=+33.831208813" watchObservedRunningTime="2026-11-03 03:00:50.670403974 +0000 UTC m=+34.840204028" Nov 03 03:00:50.975204 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:50.975158 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j76qm"] Nov 03 03:00:50.980326 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:50.980298 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68969cec_a620_4bd5_bfd1_cd203d0822b8.slice/crio-6431a15c21003bfdfc92e661c07df7bb1c19ef33b4c133a3fe262fa8b5d38edd WatchSource:0}: Error finding container 6431a15c21003bfdfc92e661c07df7bb1c19ef33b4c133a3fe262fa8b5d38edd: Status 404 returned error can't find the container with id 6431a15c21003bfdfc92e661c07df7bb1c19ef33b4c133a3fe262fa8b5d38edd Nov 03 03:00:51.082475 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.082429 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ldf8b"] Nov 03 03:00:51.085944 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:51.085904 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8aed8e_c303_437e_9e6e_37d9334a1541.slice/crio-1e937f5ba808407fbecfdf154e63630037b36979618c6a72d6be5b77718ba206 WatchSource:0}: Error finding container 1e937f5ba808407fbecfdf154e63630037b36979618c6a72d6be5b77718ba206: Status 404 returned error can't find the container with id 1e937f5ba808407fbecfdf154e63630037b36979618c6a72d6be5b77718ba206 Nov 03 03:00:51.551869 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.551792 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/1.log" Nov 03 03:00:51.552351 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.552219 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/0.log" Nov 03 03:00:51.552351 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.552255 2563 generic.go:356] "Generic (PLEG): container finished" podID="297e0916-b3bb-4002-810c-390a724cd782" containerID="356603d2bfbfe8fd2bbc1cef62804bfef5c24cb0dffeccdf9f9992efffa1f3c1" exitCode=255 Nov 03 03:00:51.552489 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.552342 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" event={"ID":"297e0916-b3bb-4002-810c-390a724cd782","Type":"ContainerDied","Data":"356603d2bfbfe8fd2bbc1cef62804bfef5c24cb0dffeccdf9f9992efffa1f3c1"} Nov 03 03:00:51.552489 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.552408 2563 scope.go:122] "RemoveContainer" containerID="cf4f46f1a3dd96cde804654cda0b893ca6d5c4cbc1714e9546b64b4ec8a46c0f" Nov 03 03:00:51.553559 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.552726 2563 scope.go:122] "RemoveContainer" containerID="356603d2bfbfe8fd2bbc1cef62804bfef5c24cb0dffeccdf9f9992efffa1f3c1" Nov 03 03:00:51.553559 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:51.553160 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-f869c_openshift-console-operator(297e0916-b3bb-4002-810c-390a724cd782)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" Nov 03 03:00:51.553963 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.553933 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ldf8b" event={"ID":"4b8aed8e-c303-437e-9e6e-37d9334a1541","Type":"ContainerStarted","Data":"1e937f5ba808407fbecfdf154e63630037b36979618c6a72d6be5b77718ba206"} Nov 03 03:00:51.556296 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:51.556272 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j76qm" event={"ID":"68969cec-a620-4bd5-bfd1-cd203d0822b8","Type":"ContainerStarted","Data":"6431a15c21003bfdfc92e661c07df7bb1c19ef33b4c133a3fe262fa8b5d38edd"} Nov 03 03:00:52.561866 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.561825 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2hdcl" event={"ID":"8fa09e76-0e81-4449-a649-1b5705b8d5bd","Type":"ContainerStarted","Data":"16d7111d2c498927c9f37e78a4b2bd9ea841944edb3f6ed7a2c06deb795ed168"} Nov 03 03:00:52.563920 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.563899 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/1.log" Nov 03 03:00:52.564237 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.564218 2563 scope.go:122] "RemoveContainer" containerID="356603d2bfbfe8fd2bbc1cef62804bfef5c24cb0dffeccdf9f9992efffa1f3c1" Nov 03 03:00:52.564413 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:52.564391 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-f869c_openshift-console-operator(297e0916-b3bb-4002-810c-390a724cd782)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" Nov 03 03:00:52.681510 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.681462 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-2hdcl" podStartSLOduration=375.422954532 podStartE2EDuration="6m20.681429647s" totalImagesPullingTime="5.258475115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:52.68006158 +0000 UTC m=+36.849861654" watchObservedRunningTime="2026-11-03 03:00:52.681429647 +0000 UTC m=+36.851229707" Nov 03 03:00:52.918471 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.918362 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-762ml"] Nov 03 03:00:52.942786 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.942756 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:52.962389 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.962354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:52.962560 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.962514 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9644b939-f36f-49ec-8388-c0e2c5bdce5a-crio-socket\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:52.962683 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.962664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gk4\" (UniqueName: \"kubernetes.io/projected/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-api-access-h6gk4\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:52.962733 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.962717 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:52.962784 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.962770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9644b939-f36f-49ec-8388-c0e2c5bdce5a-data-volume\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:52.984990 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.984956 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-762ml"] Nov 03 03:00:52.996558 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.996434 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fz9pb\"" Nov 03 03:00:52.997830 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:52.997804 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 03:00:53.063226 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.063189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.063398 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.063238 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9644b939-f36f-49ec-8388-c0e2c5bdce5a-crio-socket\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.063398 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.063330 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gk4\" (UniqueName: \"kubernetes.io/projected/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-api-access-h6gk4\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.063398 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.063364 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.063589 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.063407 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9644b939-f36f-49ec-8388-c0e2c5bdce5a-data-volume\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.063589 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.063425 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9644b939-f36f-49ec-8388-c0e2c5bdce5a-crio-socket\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.063589 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.063487 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 03:00:53.063589 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.063548 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.563531684 +0000 UTC m=+37.733331744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : configmap "kube-rbac-proxy" not found Nov 03 03:00:53.063994 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.063965 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9644b939-f36f-49ec-8388-c0e2c5bdce5a-data-volume\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.096878 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.096840 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 03:00:53.104135 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.104099 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:00:53.104303 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.104198 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:53.604174369 +0000 UTC m=+37.773974430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:00:53.132940 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.132909 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gk4\" (UniqueName: \"kubernetes.io/projected/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-api-access-h6gk4\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.190420 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.190337 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-pr56b-bzv26"] Nov 03 03:00:53.211929 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.211893 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.252955 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.252803 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pr56b-bzv26"] Nov 03 03:00:53.265345 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.265313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.265345 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.265348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.265559 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.265375 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hr7\" (UniqueName: \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.265559 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.265478 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.265675 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.265659 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vbzfk\"" Nov 03 03:00:53.366996 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.366958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.367185 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.367098 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.367185 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.367131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.367185 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.367161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hr7\" (UniqueName: \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.367951 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.367925 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.368224 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.368200 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.368488 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.368466 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.428487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.428461 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hr7\" (UniqueName: \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.534791 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.534717 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:00:53.570093 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570152 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570209 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570235 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570252 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570303 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570319 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675ff945d6-2qfxd: secret "image-registry-tls" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570303 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570284607 +0000 UTC m=+45.740084645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570235 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570350 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570359 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570381 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570369698 +0000 UTC m=+45.740169741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570400 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls podName:f2b7b1fa-14b8-47ee-a3ce-dd236209b96b nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570390632 +0000 UTC m=+45.740190676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls") pod "dns-default-dwkvw" (UID: "f2b7b1fa-14b8-47ee-a3ce-dd236209b96b") : secret "dns-default-metrics-tls" not found Nov 03 03:00:53.570495 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570426 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls podName:2469c20c-30a7-41bb-b726-d62d9324f693 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570417288 +0000 UTC m=+45.740217332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t2pr" (UID: "2469c20c-30a7-41bb-b726-d62d9324f693") : secret "samples-operator-tls" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570461 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls podName:fef674c6-31e4-4f47-b723-19bc2b38502c nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570435313 +0000 UTC m=+45.740235360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-p7hdq" (UID: "fef674c6-31e4-4f47-b723-19bc2b38502c") : secret "cluster-monitoring-operator-tls" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570478 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls podName:9db36b8c-a3dd-4bb0-8710-1d0c09e981cd nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570471609 +0000 UTC m=+45.740271647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls") pod "image-registry-675ff945d6-2qfxd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd") : secret "image-registry-tls" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570499 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570490901 +0000 UTC m=+45.740290940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570550 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.570580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570670 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570706 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:54.570698396 +0000 UTC m=+38.740498433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : configmap "kube-rbac-proxy" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570704 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570723 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7f94cf9-nvxkf: secret "image-registry-tls" not found Nov 03 03:00:53.571118 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.570770 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls podName:72608a5a-a7a0-43e7-a9c0-bc681d988262 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:01.570751923 +0000 UTC m=+45.740551975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls") pod "image-registry-5bf7f94cf9-nvxkf" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262") : secret "image-registry-tls" not found Nov 03 03:00:53.675525 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.672233 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:53.675525 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.673113 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:00:53.675525 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:53.673195 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:54.673175602 +0000 UTC m=+38.842975657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:00:53.933364 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:53.933334 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-pr56b-bzv26"] Nov 03 03:00:53.935124 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:53.935093 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4667fe_39d4_430d_8fd4_d96b5288d9d8.slice/crio-b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a WatchSource:0}: Error finding container b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a: Status 404 returned error can't find the container with id b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a Nov 03 03:00:54.573755 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:54.573716 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" event={"ID":"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a","Type":"ContainerStarted","Data":"a1541af6bd3d2b03eee1f7eb605e562f2d80c8a6ad39dab088f6f389c4af8906"} Nov 03 03:00:54.575800 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:54.575765 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pr56b-bzv26" event={"ID":"9c4667fe-39d4-430d-8fd4-d96b5288d9d8","Type":"ContainerStarted","Data":"7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99"} Nov 03 03:00:54.575917 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:54.575812 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pr56b-bzv26" event={"ID":"9c4667fe-39d4-430d-8fd4-d96b5288d9d8","Type":"ContainerStarted","Data":"b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a"} Nov 03 03:00:54.582956 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:54.582922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:54.583110 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:54.583064 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 03:00:54.583177 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:54.583135 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:56.583120161 +0000 UTC m=+40.752920199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : configmap "kube-rbac-proxy" not found Nov 03 03:00:54.683842 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:54.683805 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:54.684052 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:54.683929 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:00:54.684052 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:54.684005 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:56.683985001 +0000 UTC m=+40.853785040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:00:54.691577 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:54.691513 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" podStartSLOduration=374.454225063 podStartE2EDuration="6m21.69149175s" totalImagesPullingTime="7.237266687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:54.690732856 +0000 UTC m=+38.860532945" watchObservedRunningTime="2026-11-03 03:00:54.69149175 +0000 UTC m=+38.861291813" Nov 03 03:00:54.808495 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:54.808410 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-pr56b-bzv26" podStartSLOduration=1.8083922110000001 podStartE2EDuration="1.808392211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:54.806508446 +0000 UTC m=+38.976308535" watchObservedRunningTime="2026-11-03 03:00:54.808392211 +0000 UTC m=+38.978192270" Nov 03 03:00:55.712267 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.712231 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g"] Nov 03 03:00:55.731613 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.731581 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" Nov 03 03:00:55.736027 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.735220 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr"] Nov 03 03:00:55.750027 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.749993 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" Nov 03 03:00:55.776779 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.776742 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g"] Nov 03 03:00:55.796346 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.796304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxw5\" (UniqueName: \"kubernetes.io/projected/e9b9e6bc-874c-4111-adfc-f3c1d50f750a-kube-api-access-8pxw5\") pod \"migrator-66df5d496b-gcxsr\" (UID: \"e9b9e6bc-874c-4111-adfc-f3c1d50f750a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" Nov 03 03:00:55.796717 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.796570 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljqtl\" (UniqueName: \"kubernetes.io/projected/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96-kube-api-access-ljqtl\") pod \"migrator-66df5d496b-cgn4g\" (UID: \"6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" Nov 03 03:00:55.809478 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.809430 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 03:00:55.817928 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.817892 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 03:00:55.828896 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.828864 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-srwkl\"" Nov 03 03:00:55.843037 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.842995 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr"] Nov 03 03:00:55.897897 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.897860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxw5\" (UniqueName: \"kubernetes.io/projected/e9b9e6bc-874c-4111-adfc-f3c1d50f750a-kube-api-access-8pxw5\") pod \"migrator-66df5d496b-gcxsr\" (UID: \"e9b9e6bc-874c-4111-adfc-f3c1d50f750a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" Nov 03 03:00:55.898095 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.898036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljqtl\" (UniqueName: \"kubernetes.io/projected/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96-kube-api-access-ljqtl\") pod \"migrator-66df5d496b-cgn4g\" (UID: \"6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" Nov 03 03:00:55.966906 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.966840 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxw5\" (UniqueName: \"kubernetes.io/projected/e9b9e6bc-874c-4111-adfc-f3c1d50f750a-kube-api-access-8pxw5\") pod \"migrator-66df5d496b-gcxsr\" (UID: \"e9b9e6bc-874c-4111-adfc-f3c1d50f750a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" Nov 03 03:00:55.967519 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:55.967494 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljqtl\" (UniqueName: \"kubernetes.io/projected/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96-kube-api-access-ljqtl\") pod \"migrator-66df5d496b-cgn4g\" (UID: \"6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" Nov 03 03:00:56.066612 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.066573 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:56.066789 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.066655 2563 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:00:56.067146 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.067130 2563 scope.go:122] "RemoveContainer" containerID="356603d2bfbfe8fd2bbc1cef62804bfef5c24cb0dffeccdf9f9992efffa1f3c1" Nov 03 03:00:56.067378 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:56.067352 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-f869c_openshift-console-operator(297e0916-b3bb-4002-810c-390a724cd782)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" Nov 03 03:00:56.078313 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.078289 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" Nov 03 03:00:56.096541 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.096513 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" Nov 03 03:00:56.507558 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.507482 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g"] Nov 03 03:00:56.509190 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:00:56.509164 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b6983_cfe0_45f7_bc69_e3cb91d0ff96.slice/crio-9ac40e137e90c5074c15bdbe1350a542c90e9a07afc2a27713bd22c07a618f91 WatchSource:0}: Error finding container 9ac40e137e90c5074c15bdbe1350a542c90e9a07afc2a27713bd22c07a618f91: Status 404 returned error can't find the container with id 9ac40e137e90c5074c15bdbe1350a542c90e9a07afc2a27713bd22c07a618f91 Nov 03 03:00:56.542189 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.542156 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr"] Nov 03 03:00:56.585427 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.585384 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" event={"ID":"f58a099e-243b-4581-bebc-8ceaf23509c4","Type":"ContainerStarted","Data":"398c4c3c4436920fb52405390b9f34b68ebbddfc7a67f6f66c6597654d7464bf"} Nov 03 03:00:56.586695 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.586659 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" event={"ID":"e9b9e6bc-874c-4111-adfc-f3c1d50f750a","Type":"ContainerStarted","Data":"c5179e992165225e62a60ad17dd5d548213cf691a839bfe8b191c7b33d326771"} Nov 03 03:00:56.588031 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.587978 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" event={"ID":"6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96","Type":"ContainerStarted","Data":"9ac40e137e90c5074c15bdbe1350a542c90e9a07afc2a27713bd22c07a618f91"} Nov 03 03:00:56.605950 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.605917 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:56.606132 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:56.606063 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 03:00:56.606132 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:56.606131 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:00.606113603 +0000 UTC m=+44.775913647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : configmap "kube-rbac-proxy" not found Nov 03 03:00:56.707187 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:56.707149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:00:56.707373 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:56.707302 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:00:56.707373 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:00:56.707369 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:00.707350154 +0000 UTC m=+44.877150211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:00:57.594986 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:57.594942 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lctfx" event={"ID":"3fc36c84-f2e0-4f30-b74e-73d169c91ca8","Type":"ContainerStarted","Data":"f92d6fde44d40b75bc2b63cef536e6c3c68aeb231b39bbe500ba34faba9987cf"} Nov 03 03:00:57.713305 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:57.711775 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" podStartSLOduration=373.781794291 podStartE2EDuration="6m22.711755058s" totalImagesPullingTime="8.929960767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:56.704691353 +0000 UTC m=+40.874491432" watchObservedRunningTime="2026-11-03 03:00:57.711755058 +0000 UTC m=+41.881555119" Nov 03 03:00:57.713305 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:57.711897 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lctfx" podStartSLOduration=2.413177842 podStartE2EDuration="12.711887709s" totalImagesPullingTime="10.298709867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:57.711158427 +0000 UTC m=+41.880958504" watchObservedRunningTime="2026-11-03 03:00:57.711887709 +0000 UTC m=+41.881687770" Nov 03 03:00:58.562799 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:58.562771 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dtzkv_b70536d4-1f3e-456d-9c5c-63f25de2a77f/dns-node-resolver/0.log" Nov 03 03:00:58.599810 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:58.599771 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" event={"ID":"59642d89-3475-44f2-8ddd-c95da8e30f44","Type":"ContainerStarted","Data":"ecaadb79349702f5229ecb8ebe32ec04e10f77772d841dc7ab90ede64f4dcc5f"} Nov 03 03:00:58.716790 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:58.716734 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" podStartSLOduration=374.203246951 podStartE2EDuration="6m25.716718929s" totalImagesPullingTime="11.513471978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:58.71561909 +0000 UTC m=+42.885419149" watchObservedRunningTime="2026-11-03 03:00:58.716718929 +0000 UTC m=+42.886518989" Nov 03 03:00:59.604166 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:59.604130 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ldf8b" event={"ID":"4b8aed8e-c303-437e-9e6e-37d9334a1541","Type":"ContainerStarted","Data":"4b7073d50f7340977d1d4a43f0bf1b5a3fae1f09ea9c96276ed80c4601d8fa75"} Nov 03 03:00:59.604563 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:59.604237 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:00:59.721103 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:59.721050 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ldf8b" podStartSLOduration=35.822797835 podStartE2EDuration="43.721034751s" totalImagesPullingTime="7.898236916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:59.719678896 +0000 UTC m=+43.889478958" watchObservedRunningTime="2026-11-03 03:00:59.721034751 +0000 UTC m=+43.890834867" Nov 03 03:00:59.764202 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:00:59.764177 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnrpn_517134a0-0ad0-46d8-a99b-ed28cce40ae5/node-ca/0.log" Nov 03 03:01:00.163854 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:00.163822 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lctfx_3fc36c84-f2e0-4f30-b74e-73d169c91ca8/serve-healthcheck-canary/0.log" Nov 03 03:01:00.611184 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:00.611147 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" event={"ID":"e9b9e6bc-874c-4111-adfc-f3c1d50f750a","Type":"ContainerStarted","Data":"2c9ce813df38b5e70e16006800a41be859e763a7b8fe933c1370733a83bc8495"} Nov 03 03:01:00.652829 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:00.652788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:00.653313 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:00.653294 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:00.754192 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:00.754152 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:00.754370 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:00.754344 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:01:00.754430 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:00.754418 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:08.754399453 +0000 UTC m=+52.924199508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:01:00.773489 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:00.773428 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-gph9f_6b7adb9e-2bb1-47f2-a364-04d5ec990c3a/kube-storage-version-migrator-operator/0.log" Nov 03 03:01:01.087250 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.087157 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" podStartSLOduration=2.896137229 podStartE2EDuration="6.087139451s" totalImagesPullingTime="3.191002222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:00.729546774 +0000 UTC m=+44.899346853" watchObservedRunningTime="2026-11-03 03:01:01.087139451 +0000 UTC m=+45.256939512" Nov 03 03:01:01.088486 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.088461 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4lnbs"] Nov 03 03:01:01.109487 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.109456 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.157613 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.157562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcp5\" (UniqueName: \"kubernetes.io/projected/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-kube-api-access-sdcp5\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.158111 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.157788 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-key\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.158257 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.158237 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-cabundle\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.159165 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.159140 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4lnbs"] Nov 03 03:01:01.207422 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.207383 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 03:01:01.207791 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.207772 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8fw8c\"" Nov 03 03:01:01.207863 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.207852 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 03:01:01.212148 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.212104 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 03:01:01.212319 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.212159 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 03:01:01.259053 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.259017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcp5\" (UniqueName: \"kubernetes.io/projected/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-kube-api-access-sdcp5\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.259229 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.259077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-key\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.259278 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.259263 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-cabundle\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.260005 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.259956 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-cabundle\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.262042 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.262020 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-key\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.316889 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.316854 2563 scope.go:122] "RemoveContainer" containerID="356603d2bfbfe8fd2bbc1cef62804bfef5c24cb0dffeccdf9f9992efffa1f3c1" Nov 03 03:01:01.324654 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.324633 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcp5\" (UniqueName: \"kubernetes.io/projected/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-kube-api-access-sdcp5\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.436209 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.436173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:01.615989 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.615918 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/2.log" Nov 03 03:01:01.616497 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.616457 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/1.log" Nov 03 03:01:01.616537 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.616506 2563 generic.go:356] "Generic (PLEG): container finished" podID="297e0916-b3bb-4002-810c-390a724cd782" containerID="e08d108d1e4c3e865addb1011469ffd6b7439dd514415798897fcd17f9adbaf6" exitCode=255 Nov 03 03:01:01.616634 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.616609 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" event={"ID":"297e0916-b3bb-4002-810c-390a724cd782","Type":"ContainerDied","Data":"e08d108d1e4c3e865addb1011469ffd6b7439dd514415798897fcd17f9adbaf6"} Nov 03 03:01:01.616684 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.616673 2563 scope.go:122] "RemoveContainer" containerID="356603d2bfbfe8fd2bbc1cef62804bfef5c24cb0dffeccdf9f9992efffa1f3c1" Nov 03 03:01:01.617164 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.617125 2563 scope.go:122] "RemoveContainer" containerID="e08d108d1e4c3e865addb1011469ffd6b7439dd514415798897fcd17f9adbaf6" Nov 03 03:01:01.617350 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.617329 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-f869c_openshift-console-operator(297e0916-b3bb-4002-810c-390a724cd782)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" Nov 03 03:01:01.664596 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:01.664776 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664618 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:01.664776 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:01.664776 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664740 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:01:01.664776 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664765 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675ff945d6-2qfxd: secret "image-registry-tls" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664786 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664802 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bf7f94cf9-nvxkf: secret "image-registry-tls" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664806 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664832 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls podName:9db36b8c-a3dd-4bb0-8710-1d0c09e981cd nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.66481137 +0000 UTC m=+61.834611413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls") pod "image-registry-675ff945d6-2qfxd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd") : secret "image-registry-tls" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664856 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.664839666 +0000 UTC m=+61.834639709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664876 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls podName:72608a5a-a7a0-43e7-a9c0-bc681d988262 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.66486762 +0000 UTC m=+61.834667661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls") pod "image-registry-5bf7f94cf9-nvxkf" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262") : secret "image-registry-tls" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664877 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.664910 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.6648997 +0000 UTC m=+61.834699755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:01.664964 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.664994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.665045 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.665036212 +0000 UTC m=+61.834836276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.665097 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.665131 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls podName:fef674c6-31e4-4f47-b723-19bc2b38502c nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.665121383 +0000 UTC m=+61.834921435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-p7hdq" (UID: "fef674c6-31e4-4f47-b723-19bc2b38502c") : secret "cluster-monitoring-operator-tls" not found Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.665099 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.665148 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.665163 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls podName:f2b7b1fa-14b8-47ee-a3ce-dd236209b96b nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.665154668 +0000 UTC m=+61.834954710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls") pod "dns-default-dwkvw" (UID: "f2b7b1fa-14b8-47ee-a3ce-dd236209b96b") : secret "dns-default-metrics-tls" not found Nov 03 03:01:01.665588 ip-10-0-138-120 kubenswrapper[2563]: E1103 03:01:01.665181 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls podName:2469c20c-30a7-41bb-b726-d62d9324f693 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:17.665169709 +0000 UTC m=+61.834969753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6t2pr" (UID: "2469c20c-30a7-41bb-b726-d62d9324f693") : secret "samples-operator-tls" not found Nov 03 03:01:01.855803 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:01.855771 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4lnbs"] Nov 03 03:01:01.858292 ip-10-0-138-120 kubenswrapper[2563]: W1103 03:01:01.858261 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f2ca1a_b6e5_4a67_bb0c_e58bc8fa1feb.slice/crio-b45162c05fa2d8ae6b6f5b28a09a79e90d71fca7aaad6911ec61031f002cb14b WatchSource:0}: Error finding container b45162c05fa2d8ae6b6f5b28a09a79e90d71fca7aaad6911ec61031f002cb14b: Status 404 returned error can't find the container with id b45162c05fa2d8ae6b6f5b28a09a79e90d71fca7aaad6911ec61031f002cb14b Nov 03 03:01:02.622104 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:02.622070 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" event={"ID":"6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96","Type":"ContainerStarted","Data":"05b0de6daff1ad1b0617c2926ffc08bc2ffb1dd176b4acda200b199ba9700d64"} Nov 03 03:01:02.624127 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:02.624092 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" event={"ID":"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb","Type":"ContainerStarted","Data":"fde09d29a3f6a24d76c9ef8d317f729ee1a6f90586da8e0ff8814349ab8db6e7"} Nov 03 03:01:02.624259 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:02.624141 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" event={"ID":"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb","Type":"ContainerStarted","Data":"b45162c05fa2d8ae6b6f5b28a09a79e90d71fca7aaad6911ec61031f002cb14b"} Nov 03 03:01:02.625553 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:02.625534 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/2.log" Nov 03 03:01:02.866768 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:02.866709 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" podStartSLOduration=2.493336426 podStartE2EDuration="7.866688371s" totalImagesPullingTime="5.373351945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:02.747912728 +0000 UTC m=+46.917712810" watchObservedRunningTime="2026-11-03 03:01:02.866688371 +0000 UTC m=+47.036488427" Nov 03 03:01:02.867582 ip-10-0-138-120 kubenswrapper[2563]: I1103 03:01:02.867554 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" podStartSLOduration=1.8675434050000002 podStartE2EDuration="1.867543405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:02.865700195 +0000 UTC m=+47.035500273" watchObservedRunningTime="2026-11-03 03:01:02.867543405 +0000 UTC m=+47.037343465" Nov 03 03:01:04.904147 ip-10-0-138-120 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 03:01:04.918857 ip-10-0-138-120 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 03:01:04.919033 ip-10-0-138-120 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 03:01:04.919083 ip-10-0-138-120 systemd[1]: kubelet.service: Consumed 3.612s CPU time. Nov 03 03:01:04.935977 ip-10-0-138-120 systemd[1]: Starting Kubernetes Kubelet... Nov 03 03:01:04.998848 ip-10-0-138-120 kubenswrapper[5117]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:01:04.998848 ip-10-0-138-120 kubenswrapper[5117]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 03:01:04.998848 ip-10-0-138-120 kubenswrapper[5117]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:01:04.998848 ip-10-0-138-120 kubenswrapper[5117]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 03:01:05.001917 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001819 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.001917 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001914 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.001917 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001917 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.001917 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001920 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.001917 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001923 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.001917 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001927 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001930 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001934 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001940 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001943 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001946 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001949 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001952 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001954 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001957 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001959 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001962 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001965 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001967 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001970 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001972 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001975 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001977 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001980 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.002160 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001982 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001984 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001987 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001990 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001992 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001995 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.001997 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002001 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002004 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002006 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002009 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002011 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002014 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002016 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002019 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002021 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002024 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002026 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002029 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.002629 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002031 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002034 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002036 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002038 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002052 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002055 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002057 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002067 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002070 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002073 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002077 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002079 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002082 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002084 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002087 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002089 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002092 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002095 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002098 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002100 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.003092 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002103 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002107 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002109 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002112 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002116 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002120 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002123 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002126 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002128 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002131 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002133 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002136 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002138 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002141 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002143 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002145 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002148 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002150 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002153 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.003603 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002155 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002163 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002166 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002169 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002172 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002175 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002177 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002180 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002182 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002188 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002191 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002195 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002197 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002199 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002202 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002204 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002207 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002209 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002212 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002214 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.004089 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002217 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002219 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002222 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.002224 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003066 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003074 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003077 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003080 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003083 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003086 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003088 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003091 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003094 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003097 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003106 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003109 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003112 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003115 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003117 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.004596 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003120 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003122 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003129 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003132 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003134 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003137 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003139 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003142 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003144 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003147 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003149 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003152 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003154 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003157 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003159 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003162 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003164 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003167 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003169 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003172 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.005091 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003174 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003177 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003179 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003182 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003184 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003187 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003189 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003192 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003202 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003207 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003210 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003212 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003215 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003217 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003221 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003224 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003226 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003229 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003231 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.005616 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003234 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003236 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003239 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003241 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003243 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003246 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003248 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003251 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003253 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003256 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003258 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003261 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003263 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003266 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003268 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003270 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003273 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003275 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003278 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003280 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.006081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003283 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003286 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003294 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003297 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003299 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003302 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003304 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003308 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003311 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003313 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003315 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003318 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003320 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003322 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003325 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003327 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003329 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003332 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003334 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.006621 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003337 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003339 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003343 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003347 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003349 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003352 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003357 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003359 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003362 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003364 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003367 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003369 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003372 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003926 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003932 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003935 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003939 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003942 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003944 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003946 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.007070 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003951 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003953 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003956 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003958 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003961 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003964 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003966 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003969 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003972 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003975 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003977 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003979 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003982 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003985 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003987 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003989 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003992 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003994 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003996 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.007552 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.003999 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004002 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004005 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004007 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004010 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004013 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004021 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004025 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004027 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004031 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004033 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004036 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004038 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004042 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004045 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004056 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004060 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004063 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004065 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004068 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.008548 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004071 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004073 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004076 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004079 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004081 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004084 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004087 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004090 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004093 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004096 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004098 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004101 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004104 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004106 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004109 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004111 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004113 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004116 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004119 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004121 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.009037 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004124 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004126 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004129 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004132 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004134 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004137 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004140 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004142 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004146 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004150 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004152 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004155 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004157 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004160 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004162 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004164 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004167 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004169 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004171 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.009532 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004174 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004178 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004181 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004183 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004186 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004188 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004191 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004193 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004196 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004198 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004201 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004203 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004206 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004208 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004211 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004213 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004216 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004219 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004221 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.010012 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004224 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004226 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004324 5117 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004338 5117 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004352 5117 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004357 5117 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004361 5117 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004364 5117 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004369 5117 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004377 5117 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004380 5117 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004384 5117 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004387 5117 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004390 5117 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004395 5117 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004398 5117 flags.go:64] FLAG: --cgroup-root="" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004401 5117 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004404 5117 flags.go:64] FLAG: --client-ca-file="" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004407 5117 flags.go:64] FLAG: --cloud-provider="external" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004410 5117 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004415 5117 flags.go:64] FLAG: --cluster-domain="" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004418 5117 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004421 5117 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004424 5117 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 03:01:05.010540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004427 5117 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004431 5117 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004434 5117 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004437 5117 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004453 5117 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004456 5117 flags.go:64] FLAG: --contention-profiling="false" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004461 5117 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004464 5117 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004467 5117 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004470 5117 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004474 5117 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004477 5117 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004480 5117 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004483 5117 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004485 5117 flags.go:64] FLAG: --enable-server="true" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004488 5117 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004492 5117 flags.go:64] FLAG: --event-burst="100" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004495 5117 flags.go:64] FLAG: --event-qps="50" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004498 5117 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004501 5117 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004504 5117 flags.go:64] FLAG: --eviction-hard="" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004509 5117 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004513 5117 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004516 5117 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004519 5117 flags.go:64] FLAG: --eviction-soft="" Nov 03 03:01:05.011140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004522 5117 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004525 5117 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004528 5117 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004531 5117 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004533 5117 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004536 5117 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004539 5117 flags.go:64] FLAG: --feature-gates="" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004543 5117 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004546 5117 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004549 5117 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004552 5117 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004555 5117 flags.go:64] FLAG: --healthz-port="10248" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004558 5117 flags.go:64] FLAG: --help="false" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004561 5117 flags.go:64] FLAG: --hostname-override="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004564 5117 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004567 5117 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004570 5117 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004573 5117 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004577 5117 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004579 5117 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004582 5117 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004585 5117 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004587 5117 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 03:01:05.011752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004590 5117 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004593 5117 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004596 5117 flags.go:64] FLAG: --kube-reserved="" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004599 5117 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004602 5117 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004605 5117 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004608 5117 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004612 5117 flags.go:64] FLAG: --lock-file="" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004614 5117 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004617 5117 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004620 5117 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004627 5117 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004630 5117 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004633 5117 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004636 5117 flags.go:64] FLAG: --logging-format="text" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004638 5117 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004642 5117 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004644 5117 flags.go:64] FLAG: --manifest-url="" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004647 5117 flags.go:64] FLAG: --manifest-url-header="" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004651 5117 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004654 5117 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004658 5117 flags.go:64] FLAG: --max-pods="110" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004661 5117 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004664 5117 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004667 5117 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 03:01:05.012426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004670 5117 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004673 5117 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004676 5117 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004679 5117 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004686 5117 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004689 5117 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004691 5117 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004694 5117 flags.go:64] FLAG: --pod-cidr="" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004697 5117 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004700 5117 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004703 5117 flags.go:64] FLAG: --pods-per-core="0" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004706 5117 flags.go:64] FLAG: --port="10250" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004709 5117 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004712 5117 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a4982af9fb55fa0e" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004715 5117 flags.go:64] FLAG: --qos-reserved="" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004718 5117 flags.go:64] FLAG: --read-only-port="10255" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004720 5117 flags.go:64] FLAG: --register-node="true" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004723 5117 flags.go:64] FLAG: --register-with-taints="" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004728 5117 flags.go:64] FLAG: --registry-burst="10" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004731 5117 flags.go:64] FLAG: --registry-qps="5" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004734 5117 flags.go:64] FLAG: --reserved-cpus="" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004737 5117 flags.go:64] FLAG: --reserved-memory="" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004741 5117 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004744 5117 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004747 5117 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004750 5117 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 03:01:05.013077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004753 5117 flags.go:64] FLAG: --runonce="false" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004756 5117 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004759 5117 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004762 5117 flags.go:64] FLAG: --seccomp-default="false" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004765 5117 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004768 5117 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004772 5117 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004775 5117 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004778 5117 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004780 5117 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004783 5117 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004786 5117 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004789 5117 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004792 5117 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004795 5117 flags.go:64] FLAG: --system-cgroups="" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004798 5117 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004803 5117 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004805 5117 flags.go:64] FLAG: --tls-cert-file="" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004808 5117 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004813 5117 flags.go:64] FLAG: --tls-min-version="" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004815 5117 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004818 5117 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004821 5117 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004824 5117 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004828 5117 flags.go:64] FLAG: --v="2" Nov 03 03:01:05.013737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004832 5117 flags.go:64] FLAG: --version="false" Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004837 5117 flags.go:64] FLAG: --vmodule="" Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004841 5117 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.004843 5117 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004989 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004993 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004996 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.004999 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005002 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005004 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005007 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005009 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005012 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005014 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005018 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005020 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005023 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005025 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005027 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005030 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005032 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.014331 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005035 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005037 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005040 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005042 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005045 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005047 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005050 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005052 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005055 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005057 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005061 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005064 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005066 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005069 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005071 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005074 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005076 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005079 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005082 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005085 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.015066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005088 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005090 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005093 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005096 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005098 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005101 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005104 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005106 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005110 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005115 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005118 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005121 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005124 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005126 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005129 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005131 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005134 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005136 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005139 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.015769 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005141 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005143 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005146 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005150 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005153 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005155 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005157 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005160 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005162 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005165 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005168 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005174 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005178 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005181 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005184 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005187 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005189 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005192 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005195 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.016286 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005197 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005200 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005203 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005206 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005208 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005210 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005213 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005216 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005218 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005221 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005224 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005226 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005229 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005232 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005234 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005237 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005241 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005243 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005245 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005248 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.016765 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005250 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005253 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005255 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005257 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005261 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005263 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005266 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005268 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005271 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005273 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005276 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.005300 5117 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005355 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005360 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005364 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005367 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.017289 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005370 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005373 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005375 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005378 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005381 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005383 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005386 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005389 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005391 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005393 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005396 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005398 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005402 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005405 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005407 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005410 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005412 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005415 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005417 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005420 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.017699 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005422 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005426 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005429 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005431 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005434 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005437 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005452 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005454 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005457 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005459 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005462 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005466 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005468 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005471 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005473 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005476 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005478 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005481 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005483 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005486 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.018237 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005488 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005490 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005493 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005495 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005499 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005502 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005504 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005507 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005510 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005512 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005515 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005517 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005520 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005524 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005526 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005530 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005534 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005538 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005542 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005546 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.018852 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005549 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005553 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005557 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005560 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005564 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005568 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005572 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005576 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005579 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005581 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005584 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005586 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005588 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005593 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005597 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005601 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005605 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005609 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005611 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.019371 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005614 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005617 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005620 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005622 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005625 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005628 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005632 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005634 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005637 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005639 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005642 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005644 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005647 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005649 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005652 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005656 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005659 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005662 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005667 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.019869 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005669 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005672 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005674 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.005677 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.009855 5117 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.009871 5117 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.009889 5117 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.009894 5117 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009948 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009954 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009958 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009962 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009964 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009967 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009970 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009973 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009976 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009978 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009981 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.020338 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009984 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009986 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009989 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009991 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009994 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009996 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.009999 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010001 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010004 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010006 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010008 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010011 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010014 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010016 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010019 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010022 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010025 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010027 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010031 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010033 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.020999 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010036 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010039 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010041 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010043 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010048 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010052 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010055 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010057 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010060 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010063 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010066 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010068 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010071 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010073 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010076 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010078 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010081 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010083 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010085 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.021500 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010088 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010090 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010093 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010096 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010108 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010112 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010114 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010117 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010119 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010122 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010124 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010127 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010130 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010132 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010134 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010137 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010139 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010142 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010144 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010147 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.021946 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010150 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010152 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010156 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010160 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010162 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010165 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010167 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010170 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010172 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010175 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010178 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010180 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010182 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010185 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010187 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010190 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010193 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010195 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010198 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.022430 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010200 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010203 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010205 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010208 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010210 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010212 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010215 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010218 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010221 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010223 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010226 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010228 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010231 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010233 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010235 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010238 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.022900 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010241 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.010269 5117 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010403 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010409 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010413 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010416 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010419 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010421 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010423 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010426 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010428 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010431 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010433 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010436 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010439 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010457 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.023293 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010460 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010463 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010465 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010468 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010470 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010473 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010475 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010478 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010481 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010483 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010486 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010488 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010491 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010494 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010496 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010499 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010501 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010504 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010506 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010509 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.023786 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010511 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010514 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010516 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010518 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010521 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010524 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010529 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010532 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010534 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010537 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010540 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010542 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010545 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010548 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010550 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010553 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010555 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010558 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010561 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.024577 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010563 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010566 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010568 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010571 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010573 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010576 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010578 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010580 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010583 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010585 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010588 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010590 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010593 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010595 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010597 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010600 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010602 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010605 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010610 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010613 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.025048 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010615 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010618 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010620 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010624 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010626 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010629 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010632 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010634 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010636 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010639 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010641 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010643 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010646 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010648 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010651 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010654 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010657 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010659 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010661 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010664 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.025554 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010667 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010669 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010672 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010674 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010677 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010679 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010681 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010684 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010686 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010689 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010693 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010695 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010697 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.010720 5117 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010749 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010753 5117 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 03:01:05.026067 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010756 5117 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010759 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010761 5117 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010764 5117 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010766 5117 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010769 5117 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010772 5117 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010774 5117 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010776 5117 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010780 5117 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010784 5117 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010787 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010790 5117 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010792 5117 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010794 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010797 5117 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010799 5117 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010802 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010804 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 03:01:05.026464 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010807 5117 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010809 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010812 5117 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010814 5117 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010817 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010819 5117 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010822 5117 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010824 5117 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010828 5117 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010831 5117 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010833 5117 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010836 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010838 5117 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010841 5117 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010843 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010846 5117 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010849 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010851 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010854 5117 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010856 5117 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 03:01:05.026911 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010858 5117 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010861 5117 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010864 5117 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010866 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010868 5117 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010871 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010874 5117 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010876 5117 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010879 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010884 5117 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010888 5117 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010892 5117 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010894 5117 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010897 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010899 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010902 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010904 5117 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010907 5117 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010909 5117 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 03:01:05.027459 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010911 5117 feature_gate.go:386] unrecognized feature gate: Example Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010915 5117 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010918 5117 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010921 5117 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010923 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010925 5117 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010928 5117 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010930 5117 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010933 5117 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010935 5117 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010938 5117 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010940 5117 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010942 5117 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010945 5117 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010947 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010950 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010952 5117 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010954 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010957 5117 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010963 5117 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 03:01:05.027914 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010966 5117 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010968 5117 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010972 5117 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010974 5117 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010977 5117 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010979 5117 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010982 5117 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010984 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010987 5117 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010990 5117 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010992 5117 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010994 5117 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.010997 5117 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011001 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011003 5117 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011006 5117 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011008 5117 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011011 5117 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011013 5117 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011017 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 03:01:05.028410 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011019 5117 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011022 5117 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011024 5117 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011027 5117 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011029 5117 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:05.011032 5117 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.011094 5117 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.012013 5117 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.012074 5117 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.012254 5117 server.go:1014] "Starting client certificate rotation" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.012404 5117 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.012487 5117 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 02:55:16 +0000 UTC" deadline="2028-06-30 00:24:52.720782988 +0000 UTC" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.012516 5117 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14517h23m47.708268381s" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.012709 5117 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.013776 5117 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.016532 5117 log.go:25] "Validated CRI v1 runtime API" Nov 03 03:01:05.029178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.018254 5117 log.go:25] "Validated CRI v1 image API" Nov 03 03:01:05.029594 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.019009 5117 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 03:01:05.029594 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.020946 5117 fs.go:135] Filesystem UUIDs: map[348144bb-c90c-43ba-a724-12fe1c0d4742:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c2b12452-0a8e-44c9-ae72-ac0fe1e41a82:/dev/nvme0n1p4] Nov 03 03:01:05.029668 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.020967 5117 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05f9c340ef08fe05ccae8d80c2f769fa5d71307f6bc1aa44aac440a79296e34c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05f9c340ef08fe05ccae8d80c2f769fa5d71307f6bc1aa44aac440a79296e34c/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a3f9e980982f175a5aef7c89876a2d31fc55c56e894f09cbd8f67e57bbb737a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a3f9e980982f175a5aef7c89876a2d31fc55c56e894f09cbd8f67e57bbb737a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e55be5b8a42a19bc19296ec69498e669824aa2de0bcf8864043e5128c761431/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e55be5b8a42a19bc19296ec69498e669824aa2de0bcf8864043e5128c761431/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e937f5ba808407fbecfdf154e63630037b36979618c6a72d6be5b77718ba206/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e937f5ba808407fbecfdf154e63630037b36979618c6a72d6be5b77718ba206/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31bb7507a68daa485ad5e88fe5d112729bba7534e1d0a79c4c57e2ddbbe581be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31bb7507a68daa485ad5e88fe5d112729bba7534e1d0a79c4c57e2ddbbe581be/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4070fa999057d8bd19e16e519137e585ff5901a16e261fb86c2b7a946b82aa47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4070fa999057d8bd19e16e519137e585ff5901a16e261fb86c2b7a946b82aa47/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41c27335458a2d17f09e3bef82fe1b19455ea376bc7130e1a04bc97477ee9acd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41c27335458a2d17f09e3bef82fe1b19455ea376bc7130e1a04bc97477ee9acd/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4267a533b031d43686708ddfebe0b727a362020cbaa383c778e174b715ab1dc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4267a533b031d43686708ddfebe0b727a362020cbaa383c778e174b715ab1dc6/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6431a15c21003bfdfc92e661c07df7bb1c19ef33b4c133a3fe262fa8b5d38edd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6431a15c21003bfdfc92e661c07df7bb1c19ef33b4c133a3fe262fa8b5d38edd/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/722128c275ce379f3503e2b3fe3cf26bf0d3a0b451e8b7570bbed3c240c7a593/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/722128c275ce379f3503e2b3fe3cf26bf0d3a0b451e8b7570bbed3c240c7a593/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78c9f862d266c6d3d4de307c7332aeca1b614cb3782755bb2b2530d93924756a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78c9f862d266c6d3d4de307c7332aeca1b614cb3782755bb2b2530d93924756a/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8649e252d12ca03cf941fca6ac569943d9701612bebb3f3f385ecd5e64909b5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8649e252d12ca03cf941fca6ac569943d9701612bebb3f3f385ecd5e64909b5b/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b9ba714f837281310eecb088372c941689d8434e845664f97fca911d45a5aa0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b9ba714f837281310eecb088372c941689d8434e845664f97fca911d45a5aa0/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9194453833f4e13774ca808f943858755243ffde3a9fa79e90e2aafa829a0fef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9194453833f4e13774ca808f943858755243ffde3a9fa79e90e2aafa829a0fef/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9613fd2e56776a7bfcf8d988b7c71ff9de0a6867f2f7f1befdbd2f9d1357a0f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9613fd2e56776a7bfcf8d988b7c71ff9de0a6867f2f7f1befdbd2f9d1357a0f1/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ac40e137e90c5074c15bdbe1350a542c90e9a07afc2a27713bd22c07a618f91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ac40e137e90c5074c15bdbe1350a542c90e9a07afc2a27713bd22c07a618f91/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a579d256c87d8443d30c12e6b1124b5846be4b75db4facfd54e456d212857253/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a579d256c87d8443d30c12e6b1124b5846be4b75db4facfd54e456d212857253/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9acee61234ba09bec1e361ee402c94081657d7280744d288f8140a29ed1f6c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9acee61234ba09bec1e361ee402c94081657d7280744d288f8140a29ed1f6c6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b06265fc4458ab46cac74ec59b2a8f9963c357920e1335ad783c04dea34319a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b06265fc4458ab46cac74ec59b2a8f9963c357920e1335ad783c04dea34319a8/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b45162c05fa2d8ae6b6f5b28a09a79e90d71fca7aaad6911ec61031f002cb14b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b45162c05fa2d8ae6b6f5b28a09a79e90d71fca7aaad6911ec61031f002cb14b/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0e86de06d203ddb4d647ed53a14c73f8e4cd96aef670ea2ab644f67588d7920/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0e86de06d203ddb4d647ed53a14c73f8e4cd96aef670ea2ab644f67588d7920/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5179e992165225e62a60ad17dd5d548213cf691a839bfe8b191c7b33d326771/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5179e992165225e62a60ad17dd5d548213cf691a839bfe8b191c7b33d326771/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fed7ab78d88bd0b4128b19529522d09ef1b960ab09e1046d7e2b14577bbf8584/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fed7ab78d88bd0b4128b19529522d09ef1b960ab09e1046d7e2b14577bbf8584/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb/volumes/kubernetes.io~projected/kube-api-access-sdcp5:{mountpoint:/var/lib/kubelet/pods/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb/volumes/kubernetes.io~projected/kube-api-access-sdcp5 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb/volumes/kubernetes.io~secret/signing-key major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2/volumes/kubernetes.io~projected/kube-api-access-f892q:{mountpoint:/var/lib/kubelet/pods/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2/volumes/kubernetes.io~projected/kube-api-access-f892q major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e4b8195-7644-4736-af66-d32d0d48551e/volumes/kubernetes.io~projected/kube-api-access-42lhw:{mountpoint:/var/lib/kubelet/pods/0e4b8195-7644-4736-af66-d32d0d48551e/volumes/kubernetes.io~projected/kube-api-access-42lhw major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eb3b873-057e-44cd-9ec6-f22dd6f809b2/volumes/kubernetes.io~projected/kube-api-access-6gx4h:{mountpoint:/var/lib/kubelet/pods/0eb3b873-057e-44cd-9ec6-f22dd6f809b2/volumes/kubernetes.io~projected/kube-api-access-6gx4h major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1714a90e-d9bd-4a76-bc61-295f6978bf17/volumes/kubernetes.io~projected/kube-api-access-d2xgb:{mountpoint:/var/lib/kubelet/pods/1714a90e-d9bd-4a76-bc61-295f6978bf17/volumes/kubernetes.io~projected/kube-api-access-d2xgb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19f60386-016f-4213-ad61-462deca38b28/volumes/kubernetes.io~projected/kube-api-access-stnf6:{mountpoint:/var/lib/kubelet/pods/19f60386-016f-4213-ad61-462deca38b28/volumes/kubernetes.io~projected/kube-api-access-stnf6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2469c20c-30a7-41bb-b726-d62d9324f693/volumes/kubernetes.io~projected/kube-api-access-gw5tr:{mountpoint:/var/lib/kubelet/pods/2469c20c-30a7-41bb-b726-d62d9324f693/volumes/kubernetes.io~projected/kube-api-access-gw5tr major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/297e0916-b3bb-4002-810c-390a724cd782/volumes/kubernetes.io~projected/kube-api-access-lf5fl:{mountpoint:/var/lib/kubelet/pods/297e0916-b3bb-4002-810c-390a724cd782/volumes/kubernetes.io~projected/kube-api-access-lf5fl major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/297e0916-b3bb-4002-810c-390a724cd782/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/297e0916-b3bb-4002-810c-390a724cd782/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc36c84-f2e0-4f30-b74e-73d169c91ca8/volumes/kubernetes.io~projected/kube-api-access-777xw:{mountpoint:/var/lib/kubelet/pods/3fc36c84-f2e0-4f30-b74e-73d169c91ca8/volumes/kubernetes.io~projected/kube-api-access-777xw major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc36c84-f2e0-4f30-b74e-73d169c91ca8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3fc36c84-f2e0-4f30-b74e-73d169c91ca8/volumes/kubernetes.io~secret/cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b8aed8e-c303-437e-9e6e-37d9334a1541/volumes/kubernetes.io~projected/kube-api-access-r4k7s:{mountpoint:/var/lib/kubelet/pods/4b8aed8e-c303-437e-9e6e-37d9334a1541/volumes/kubernetes.io~projected/kube-api-access-r4k7s major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf/volumes/kubernetes.io~projected/kube-api-access-hmswh:{mountpoint:/var/lib/kubelet/pods/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf/volumes/kubernetes.io~projected/kube-api-access-hmswh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50ae62be-5346-4aff-b350-305a9a5e1a6d/volumes/kubernetes.io~projected/kube-api-access-2g5w4:{mountpoint:/var/lib/kubelet/pods/50ae62be-5346-4aff-b350-305a9a5e1a6d/volumes/kubernetes.io~projected/kube-api-access-2g5w4 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/517134a0-0ad0-46d8-a99b-ed28cce40ae5/volumes/kubernetes.io~projected/kube-api-access-dn5q2:{mountpoint:/var/lib/kubelet/pods/517134a0-0ad0-46d8-a99b-ed28cce40ae5/volumes/kubernetes.io~projected/kube-api-access-dn5q2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57dbddca-a044-4478-8cfe-9867494e4291/volumes/kubernetes.io~projected/kube-api-access-dsgnt:{mountpoint:/var/lib/kubelet/pods/57dbddca-a044-4478-8cfe-9867494e4291/volumes/kubernetes.io~projected/kube-api-access-dsgnt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59642d89-3475-44f2-8ddd-c95da8e30f44/volumes/kubernetes.io~projected/kube-api-access-h4j48:{mountpoint:/var/lib/kubelet/pods/59642d89-3475-44f2-8ddd-c95da8e30f44/volumes/kubernetes.io~projected/kube-api-access-h4j48 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59642d89-3475-44f2-8ddd-c95da8e30f44/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/59642d89-3475-44f2-8ddd-c95da8e30f44/volumes/kubernetes.io~secret/serving-cert major:0 minor:199 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68969cec-a620-4bd5-bfd1-cd203d0822b8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/68969cec-a620-4bd5-bfd1-cd203d0822b8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volumes/kubernetes.io~projected/kube-api-access-9pqmg:{mountpoint:/var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volumes/kubernetes.io~projected/kube-api-access-9pqmg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a/volumes/kubernetes.io~projected/kube-api-access-5tl5c:{mountpoint:/var/lib/kubelet/pods/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a/volumes/kubernetes.io~projected/kube-api-access-5tl5c major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96/volumes/kubernetes.io~projected/kube-api-access-ljqtl:{mountpoint:/var/lib/kubelet/pods/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96/volumes/kubernetes.io~projected/kube-api-access-ljqtl major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/724f0110-6fa5-473c-a88b-c514952a50a3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/724f0110-6fa5-473c-a88b-c514952a50a3/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~projected/kube-api-access-ggrcw:{mountpoint:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~projected/kube-api-access-ggrcw major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83b404fd-1b9b-4088-ba59-d9997f5a1558/volumes/kubernetes.io~projected/kube-api-access-crsn5:{mountpoint:/var/lib/kubelet/pods/83b404fd-1b9b-4088-ba59-d9997f5a1558/volumes/kubernetes.io~projected/kube-api-access-crsn5 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa09e76-0e81-4449-a649-1b5705b8d5bd/volumes/kubernetes.io~projected/kube-api-access-qtspf:{mountpoint:/var/lib/kubelet/pods/8fa09e76-0e81-4449-a649-1b5705b8d5bd/volumes/kubernetes.io~projected/kube-api-access-qtspf major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa09e76-0e81-4449-a649-1b5705b8d5bd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8fa09e76-0e81-4449-a649-1b5705b8d5bd/volumes/kubernetes.io~secret/serving-cert major:0 minor:198 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9644b939-f36f-49ec-8388-c0e2c5bdce5a/volumes/kubernetes.io~projected/kube-api-access-h6gk4:{mountpoint:/var/lib/kubelet/pods/9644b939-f36f-49ec-8388-c0e2c5bdce5a/volumes/kubernetes.io~projected/kube-api-access-h6gk4 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c4667fe-39d4-430d-8fd4-d96b5288d9d8/volumes/kubernetes.io~projected/kube-api-access-k2hr7:{mountpoint:/var/lib/kubelet/pods/9c4667fe-39d4-430d-8fd4-d96b5288d9d8/volumes/kubernetes.io~projected/kube-api-access-k2hr7 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~projected/kube-api-access-4b8lc:{mountpoint:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~projected/kube-api-access-4b8lc major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:191 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b70536d4-1f3e-456d-9c5c-63f25de2a77f/volumes/kubernetes.io~projected/kube-api-access-fztvl:{mountpoint:/var/lib/kubelet/pods/b70536d4-1f3e-456d-9c5c-63f25de2a77f/volumes/kubernetes.io~projected/kube-api-access-fztvl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~projected/kube-api-access-gmg4d:{mountpoint:/var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~projected/kube-api-access-gmg4d major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~secret/default-certificate major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~secret/stats-auth major:0 minor:195 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9b9e6bc-874c-4111-adfc-f3c1d50f750a/volumes/kubernetes.io~projected/kube-api-access-8pxw5:{mountpoint:/var/lib/kubelet/pods/e9b9e6bc-874c-4111-adfc-f3c1d50f750a/volumes/kubernetes.io~projected/kube-api-access-8pxw5 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b/volumes/kubernetes.io~projected/kube-api-access-ktp62:{mountpoint:/var/lib/kubelet/pods/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b/volumes/kubernetes.io~projected/kube-api-access-ktp62 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f58a099e-243b-4581-bebc-8ceaf23509c4/volumes/kubernetes.io~projected/kube-api-access-c4gr2:{mountpoint:/var/lib/kubelet/pods/f58a099e-243b-4581-bebc-8ceaf23509c4/volumes/kubernetes.io~projected/kube-api-access-c4gr2 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~projected/kube-api-access-tgq6x:{mountpoint:/var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~projected/kube-api-access-tgq6x major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fef674c6-31e4-4f47-b723-19bc2b38502c/volumes/kubernetes.io~projected/kube-api-access-79gl9:{mountpoint:/var/lib/kubelet/pods/fef674c6-31e4-4f47-b723-19bc2b38502c/volumes/kubernetes.io~projected/kube-api-access-79gl9 major:0 minor:215 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/07b55c9f7f18ce1684be6e98d66a00d0849ab5881e537758c6876c4486411b2a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/339bf41041db96df1e0eda2e63bd7ac1ebe8205c1e807061de1745b4f7fdd4a3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/62d9aa355d94722fc12600fa80c8d12c859ecb40f2c87ee572c5f636f0d6858e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/474cbd1a5d171f9282726db40b4a03f60eabb2f742b0bbfdf5c639e985e8c35f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/139c7979bb5a1d8bff04198901cb0c64e8b9c9081405e2759266aea94253d272/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e0beb6e7bbcb3ee3e743330ce50b42959f64d4d1d2d45cc5d059f2ccdcfc13fb/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/17f5962d1aa6726af8e56143807685d3f11a8cc7c9867b2c2d4da9908c70c279/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/containers/storage/overlay/407ae66fbee0449c22dad1fb4bee49d571fe90174bdb8928a84772212196c3c7/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/77b9b38265e6ebe83b5d472bfa7b15d9fb413e3a6af0ce0361e24ed50d56ee20/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/51405f6303e88db4ab99eb49b37e825e70dbf866116adb909d23e13af527d190/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6c0db90316fbc2febb0f4ba9390757f5ac67df81c89b4650a27748490151b94e/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/5ab70d4f17efab3d13a6fa2062c9926b27424d3df9dd75328099eed51d584f83/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/749f47f9d19be656ca811dd9251999eebfa71bded40caa39c436ff82de5def71/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/f640197ccdb7dfa33c4f89f0c03d0849afe9628d8674834be97384a852b4cb0d/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/67645bb635ac597877706a3705b0a09de7ec1c9eb2e0becf641536a3a4726639/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/8a31307434c6da096fa6b60f5482b0c6b9589017e026c31e02e74c593436f735/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/db5ea3c5b737c5a183691d4924a224bfcce4ea659611c4bfbed65faa7385f39e/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/f871344d1d6fc9fc48f14d4d414035552eb1e4e55baa0ad6163be0bc8d63c074/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/0e3f3fcdc31ff09c35fe2f792b15d2334fde0be4a5d09d2c876fa7fa1dd225bf/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/b09b29fcfd60b397948cdfc775f8ac25212164e2d722d2dc97fafc3e822a3447/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/e3bd746ee0bef8879f8f54d1fdd7a0c8d3fd9ec40f980c713f2c2e03b696479e/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/79cf3fa30ac3131019c337b2cbc2e691e6e3ea9f63dc9e528a0d0aca65e79cd9/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/22f37cbd0a6c51eb7c515eb889a7ea6788e30b8cd7034735139a23673627dd8b/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/09a3a672c87ae6bb3262cb0f823d567b265f3bd7ecc48199db935c57139abbd3/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/ad51b6f42282e385aeefa7337a0f70086aecf92500603216ee1b86c52d3e30fc/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/35545d089d861526f5094cded2fa43b941c668b69f3b3f90dd76800c26bb3aae/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/e2ddd9aea3e03066abaf8c10b5ea8187d6d6b274bd937d405a8f57ad66521272/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/4e9b5429c8c87977f8645cce2afc6b59c62c66bbfbbbfa7cbbc06c0d50df7c74/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f506697449ff7cab5aeadef049f365dfbc0eb3420094675fe45ab2a1491bac3a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/8c574139209b07ba0d696c75486eb532283f8746c126da7a1cda96587033ef03/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/cdbaadce297e0668a9b604a946148cd192067e4cae51937b1624424fed54392f/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2975cf9de08b6e4f0fc70baf47e238d3ed81b8ef6d0d21a3937f324df3de9408/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/8ed2b1f4271e907a6ff49cf0b66373b28fb06efb6cf9405c6a6d419794f3e437/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/b494c93bcbfe9158f351fc99acc5511587cd44905413ef52c7ea9830c5f0ad18/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/f2ee5270fab2ada8f91a6ab87e4d0207761b88da77deca0e6c40ec93dfc7dc5d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/91410f2d3fa95e2ede616f30cf907443e3ebae020614e56650188f81dc4af63f/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/083d03fd0cf20de6d2cb4c6403fe3a25c1525d03c75b429fbe60d96b8d1e6f92/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/b6c96779cc58f6e2de12ba604567b914e1d01e28e9ce694025e7f27a2c1de94c/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/6d4c4b373a2466f65adcc7fa5998ea8345866609fb8e75e6941c0d728f9a431b/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/37427cc8b3eb3bd8a90a68dade563bf9412f1cf6d298c8d0d7cb36a92824fec2/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/b2a2b238d6b6784e584c22d2be0eb2675b86b9bba0ee630b6aba11f4a733cc95/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/90128b720519a980e5a07e98dd02f31862e335a5598e1918091d399ef593acb5/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/0abc0728859acf0eea611e40d1cfe2f71d894efe01c802c57b1a3bb7f07f27c7/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/fc124cadad7c66df89853eb09f6cf0618e33c47552a99e1d1e270caaba0782d0/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/0d68dfc4d3a0c497bf7fd884ecf7dfaac1d93dbc24134ee006ebdd7bb3d5c913/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/6e02aa48befc8ea53878e56916eceadf093a4b0c6352f32159802500562a959e/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/36e8e18c974699047950d244ffbd712b36a4336a3152e3048210274869505c24/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/d0acdfc251d2c837a7014a7791aac01df7312143d5d7950c28f69158206b11c9/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/a11827f9257ccfbcacfa7b5213a089b89f292931595474b174fea2132d760327/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/9aca24709204ac49097e727b6a04cfdaa87a774f6de327d940ddcb7faad94a2d/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/c155d3f9aa9ef2e5097d51b5f28732a3c88553f7236283ae46f0046c4c7f93c0/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/63d4e6f427aedefbcbd8bb45103babf51c9b5dc716b04f0e939ec9f0aa06e5cc/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/76672a10346f93fd1002cc65f859050710eb8a4f876c2e15c404d24fa591a309/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/409eef9ac269d64dc781a85b7c27e8b50d1a48882d76cfc04ae61097ab4101f8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f80b0d2ce3721f24bc87bab07acbb669efb6448d9a242c2295875e8c8d3ec5f3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a869b1e4067a5a03e99c199a0879d30eb8f94e8e6abd4e1da5291862beb0a927/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/27b208201f874f6378671060d4597f097891be6d908ae358b814b813f79066ea/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c75514060b329ff6f2c2e5f4aa6c59f455306b4d71d5eda217fd5104e7df6184/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/25d510436354c66150ef6e81d7465ee681e6d85141210fbc09dcf6f81931c611/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b594b91d441cea19108ad5f3b400a4788fdb27d3d5f71a24e5b52d08205b4660/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d518c848f7d743f475fd1b020a030647d4372a44aad7b345d43e01e97b85cca3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8a3a76389c5bac712ab0e744b0ab52df1fe36755e66e8f19489d148ee4856966/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 03:01:05.030114 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.029741 5117 manager.go:217] Machine: {Timestamp:2026-11-03 03:01:05.029261235 +0000 UTC m=+0.059859763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3123378 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec281c013d75633e3b09b61cc5b3a040 SystemUUID:ec281c01-3d75-633e-3b09-b61cc5b3a040 BootID:f94219f7-05e1-4e96-837b-6354b837721a Filesystems:[{Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9acee61234ba09bec1e361ee402c94081657d7280744d288f8140a29ed1f6c6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~projected/kube-api-access-gmg4d DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volumes/kubernetes.io~projected/kube-api-access-9pqmg DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2/volumes/kubernetes.io~projected/kube-api-access-f892q DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf/volumes/kubernetes.io~projected/kube-api-access-hmswh DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05f9c340ef08fe05ccae8d80c2f769fa5d71307f6bc1aa44aac440a79296e34c/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/fef674c6-31e4-4f47-b723-19bc2b38502c/volumes/kubernetes.io~projected/kube-api-access-79gl9 DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/297e0916-b3bb-4002-810c-390a724cd782/volumes/kubernetes.io~projected/kube-api-access-lf5fl DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0eb3b873-057e-44cd-9ec6-f22dd6f809b2/volumes/kubernetes.io~projected/kube-api-access-6gx4h DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~projected/kube-api-access-ggrcw DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e4b8195-7644-4736-af66-d32d0d48551e/volumes/kubernetes.io~projected/kube-api-access-42lhw DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa09e76-0e81-4449-a649-1b5705b8d5bd/volumes/kubernetes.io~projected/kube-api-access-qtspf DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e55be5b8a42a19bc19296ec69498e669824aa2de0bcf8864043e5128c761431/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/517134a0-0ad0-46d8-a99b-ed28cce40ae5/volumes/kubernetes.io~projected/kube-api-access-dn5q2 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2469c20c-30a7-41bb-b726-d62d9324f693/volumes/kubernetes.io~projected/kube-api-access-gw5tr DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31bb7507a68daa485ad5e88fe5d112729bba7534e1d0a79c4c57e2ddbbe581be/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6431a15c21003bfdfc92e661c07df7bb1c19ef33b4c133a3fe262fa8b5d38edd/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/57dbddca-a044-4478-8cfe-9867494e4291/volumes/kubernetes.io~projected/kube-api-access-dsgnt DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1714a90e-d9bd-4a76-bc61-295f6978bf17/volumes/kubernetes.io~projected/kube-api-access-d2xgb DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~projected/kube-api-access-4b8lc DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b45162c05fa2d8ae6b6f5b28a09a79e90d71fca7aaad6911ec61031f002cb14b/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b70536d4-1f3e-456d-9c5c-63f25de2a77f/volumes/kubernetes.io~projected/kube-api-access-fztvl DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb/volumes/kubernetes.io~projected/kube-api-access-sdcp5 DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/724f0110-6fa5-473c-a88b-c514952a50a3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa09e76-0e81-4449-a649-1b5705b8d5bd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:198 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc36c84-f2e0-4f30-b74e-73d169c91ca8/volumes/kubernetes.io~projected/kube-api-access-777xw DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f58a099e-243b-4581-bebc-8ceaf23509c4/volumes/kubernetes.io~projected/kube-api-access-c4gr2 DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4070fa999057d8bd19e16e519137e585ff5901a16e261fb86c2b7a946b82aa47/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:200 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc36c84-f2e0-4f30-b74e-73d169c91ca8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9613fd2e56776a7bfcf8d988b7c71ff9de0a6867f2f7f1befdbd2f9d1357a0f1/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b9ba714f837281310eecb088372c941689d8434e845664f97fca911d45a5aa0/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59642d89-3475-44f2-8ddd-c95da8e30f44/volumes/kubernetes.io~projected/kube-api-access-h4j48 DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c4667fe-39d4-430d-8fd4-d96b5288d9d8/volumes/kubernetes.io~projected/kube-api-access-k2hr7 DeviceMajor:0 DeviceMinor:309 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:362 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:195 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8649e252d12ca03cf941fca6ac569943d9701612bebb3f3f385ecd5e64909b5b/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ac40e137e90c5074c15bdbe1350a542c90e9a07afc2a27713bd22c07a618f91/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~projected/kube-api-access-tgq6x DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/297e0916-b3bb-4002-810c-390a724cd782/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b/volumes/kubernetes.io~projected/kube-api-access-ktp62 DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41c27335458a2d17f09e3bef82fe1b19455ea376bc7130e1a04bc97477ee9acd/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68969cec-a620-4bd5-bfd1-cd203d0822b8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5179e992165225e62a60ad17dd5d548213cf691a839bfe8b191c7b33d326771/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a/volumes/kubernetes.io~projected/kube-api-access-5tl5c DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4b8aed8e-c303-437e-9e6e-37d9334a1541/volumes/kubernetes.io~projected/kube-api-access-r4k7s DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96/volumes/kubernetes.io~projected/kube-api-access-ljqtl DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9194453833f4e13774ca808f943858755243ffde3a9fa79e90e2aafa829a0fef/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19f60386-016f-4213-ad61-462deca38b28/volumes/kubernetes.io~projected/kube-api-access-stnf6 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4267a533b031d43686708ddfebe0b727a362020cbaa383c778e174b715ab1dc6/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c02e7385-46cc-46a6-822f-f47fe2819a3f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:196 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e937f5ba808407fbecfdf154e63630037b36979618c6a72d6be5b77718ba206/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fed7ab78d88bd0b4128b19529522d09ef1b960ab09e1046d7e2b14577bbf8584/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/722128c275ce379f3503e2b3fe3cf26bf0d3a0b451e8b7570bbed3c240c7a593/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb498359-0464-4428-8a7d-4901cea017ac/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b06265fc4458ab46cac74ec59b2a8f9963c357920e1335ad783c04dea34319a8/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9644b939-f36f-49ec-8388-c0e2c5bdce5a/volumes/kubernetes.io~projected/kube-api-access-h6gk4 DeviceMajor:0 DeviceMinor:304 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9b9e6bc-874c-4111-adfc-f3c1d50f750a/volumes/kubernetes.io~projected/kube-api-access-8pxw5 DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:191 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/59642d89-3475-44f2-8ddd-c95da8e30f44/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:199 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/50ae62be-5346-4aff-b350-305a9a5e1a6d/volumes/kubernetes.io~projected/kube-api-access-2g5w4 DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/83b404fd-1b9b-4088-ba59-d9997f5a1558/volumes/kubernetes.io~projected/kube-api-access-crsn5 DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78c9f862d266c6d3d4de307c7332aeca1b614cb3782755bb2b2530d93924756a/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a3f9e980982f175a5aef7c89876a2d31fc55c56e894f09cbd8f67e57bbb737a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a579d256c87d8443d30c12e6b1124b5846be4b75db4facfd54e456d212857253/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0e86de06d203ddb4d647ed53a14c73f8e4cd96aef670ea2ab644f67588d7920/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:05f9c340ef08fe0 MacAddress:ca:8c:ae:24:07:25 Speed:10000 Mtu:8901} {Name:1e55be5b8a42a19 MacAddress:de:dd:53:b9:1c:28 Speed:10000 Mtu:8901} {Name:1e937f5ba808407 MacAddress:06:61:60:14:1b:40 Speed:10000 Mtu:8901} {Name:6431a15c21003bf MacAddress:06:48:de:4e:34:1f Speed:10000 Mtu:8901} {Name:78c9f862d266c6d MacAddress:6a:a9:c4:16:98:e5 Speed:10000 Mtu:8901} {Name:8649e252d12ca03 MacAddress:52:c6:98:cd:b6:3d Speed:10000 Mtu:8901} {Name:8b9ba714f837281 MacAddress:9e:0f:ff:db:ae:f8 Speed:10000 Mtu:8901} {Name:9194453833f4e13 MacAddress:5a:f0:aa:58:3c:06 Speed:10000 Mtu:8901} {Name:9613fd2e56776a7 MacAddress:ba:41:14:39:16:b3 Speed:10000 Mtu:8901} {Name:9ac40e137e90c50 MacAddress:de:31:48:50:1b:69 Speed:10000 Mtu:8901} {Name:a579d256c87d844 MacAddress:d6:f2:da:74:0b:e1 Speed:10000 Mtu:8901} {Name:b06265fc4458ab4 MacAddress:a6:9c:d3:c7:a5:74 Speed:10000 Mtu:8901} {Name:b2f28c2d1f3024a MacAddress:1e:34:2d:5c:07:ab Speed:10000 Mtu:8901} {Name:b45162c05fa2d8a MacAddress:3e:2e:40:a8:3c:37 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:38:bb:4a:92:65 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:da:78:7f:87:64 Speed:0 Mtu:8901} {Name:c0e86de06d203dd MacAddress:a2:65:07:9f:76:27 Speed:10000 Mtu:8901} {Name:c5179e992165225 MacAddress:5e:de:89:e9:8d:45 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:38:bb:4a:92:65 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:e3:b4:6e:23:29 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 03:01:05.030354 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030119 5117 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 03:01:05.030354 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030163 5117 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 03:01:05.030587 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030557 5117 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 03:01:05.030747 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030589 5117 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-120.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 03:01:05.030837 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030750 5117 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 03:01:05.030837 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030763 5117 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 03:01:05.030837 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030778 5117 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:01:05.030837 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030796 5117 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 03:01:05.030837 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.030821 5117 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 03:01:05.031061 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.031047 5117 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 03:01:05.031196 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.031184 5117 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 03:01:05.031259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.031233 5117 kubelet.go:493] "Attempting to sync node with API server" Nov 03 03:01:05.031259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.031247 5117 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 03:01:05.031341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.031261 5117 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 03:01:05.031341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.031273 5117 kubelet.go:405] "Adding apiserver pod source" Nov 03 03:01:05.031341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.031284 5117 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 03:01:05.033245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.033065 5117 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 03:01:05.033245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.033088 5117 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:01:05.033927 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.033900 5117 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 03:01:05.036014 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.035998 5117 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 03:01:05.036014 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036013 5117 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 03:01:05.036129 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036119 5117 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 03:01:05.036308 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036298 5117 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 03:01:05.036338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036327 5117 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 03:01:05.036424 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036415 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 03:01:05.036472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036429 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 03:01:05.036472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036435 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 03:01:05.036472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036458 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 03:01:05.036472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036467 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 03:01:05.036472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036473 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036479 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036486 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036492 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036498 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036506 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036516 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036542 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 03:01:05.036601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.036548 5117 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 03:01:05.039875 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.039864 5117 server.go:1263] "Started kubelet" Nov 03 03:01:05.039972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.039947 5117 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 03:01:05.040024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.039959 5117 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 03:01:05.040469 ip-10-0-138-120 systemd[1]: Started Kubernetes Kubelet. Nov 03 03:01:05.040583 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.040479 5117 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 03:01:05.040806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.040786 5117 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 03:01:05.041171 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.041152 5117 server.go:317] "Adding debug handlers to kubelet server" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.043426 5117 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.043958 5117 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.044115 5117 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.044128 5117 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.044267 5117 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.044467 5117 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.044896 5117 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:18 +0000 UTC" deadline="2028-05-11 15:55:46.654706432 +0000 UTC" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.044934 5117 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13332h54m41.609776664s" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.045685 5117 factory.go:153] Registering CRI-O factory Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.045705 5117 factory.go:223] Registration of the crio container factory successfully Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.045769 5117 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.045779 5117 factory.go:55] Registering systemd factory Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.045788 5117 factory.go:223] Registration of the systemd container factory successfully Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.045812 5117 factory.go:103] Registering Raw factory Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.045837 5117 manager.go:1199] Started watching for new ooms in manager Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.046303 5117 manager.go:319] Starting recovery of all containers Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049308 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="297e0916-b3bb-4002-810c-390a724cd782" volumeName="kubernetes.io/secret/297e0916-b3bb-4002-810c-390a724cd782-serving-cert" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049362 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57dbddca-a044-4478-8cfe-9867494e4291" volumeName="kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-binary-copy" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049387 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57dbddca-a044-4478-8cfe-9867494e4291" volumeName="kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049399 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa09e76-0e81-4449-a649-1b5705b8d5bd" volumeName="kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-trusted-ca-bundle" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049427 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" volumeName="kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049467 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe" volumeName="kubernetes.io/configmap/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-nginx-conf" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049483 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af" volumeName="kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-config" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049495 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9644b939-f36f-49ec-8388-c0e2c5bdce5a" volumeName="kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049525 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9644b939-f36f-49ec-8388-c0e2c5bdce5a" volumeName="kubernetes.io/projected/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-api-access-h6gk4" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049537 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" volumeName="kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049561 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" volumeName="kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049578 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c02e7385-46cc-46a6-822f-f47fe2819a3f" volumeName="kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-stats-auth" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049633 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="297e0916-b3bb-4002-810c-390a724cd782" volumeName="kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-trusted-ca" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049663 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c75a11d-6c39-4d24-ac43-0b0dc04f4faf" volumeName="kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cni-binary-copy" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049702 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af" volumeName="kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-script-lib" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049718 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa09e76-0e81-4449-a649-1b5705b8d5bd" volumeName="kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-service-ca-bundle" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049766 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9644b939-f36f-49ec-8388-c0e2c5bdce5a" volumeName="kubernetes.io/empty-dir/9644b939-f36f-49ec-8388-c0e2c5bdce5a-data-volume" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049784 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" volumeName="kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049798 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb" volumeName="kubernetes.io/secret/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-key" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049859 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57dbddca-a044-4478-8cfe-9867494e4291" volumeName="kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 03:01:05.049926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.049891 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af" volumeName="kubernetes.io/projected/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-kube-api-access-9pqmg" seLinuxMountContext="" Nov 03 03:01:05.051813 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.051781 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" volumeName="kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca" seLinuxMountContext="" Nov 03 03:01:05.051940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.051839 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb498359-0464-4428-8a7d-4901cea017ac" volumeName="kubernetes.io/projected/fb498359-0464-4428-8a7d-4901cea017ac-kube-api-access-tgq6x" seLinuxMountContext="" Nov 03 03:01:05.051940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.051889 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb3b873-057e-44cd-9ec6-f22dd6f809b2" volumeName="kubernetes.io/configmap/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-iptables-alerter-script" seLinuxMountContext="" Nov 03 03:01:05.051940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.051905 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="517134a0-0ad0-46d8-a99b-ed28cce40ae5" volumeName="kubernetes.io/projected/517134a0-0ad0-46d8-a99b-ed28cce40ae5-kube-api-access-dn5q2" seLinuxMountContext="" Nov 03 03:01:05.051940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.051936 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7adb9e-2bb1-47f2-a364-04d5ec990c3a" volumeName="kubernetes.io/projected/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-kube-api-access-5tl5c" seLinuxMountContext="" Nov 03 03:01:05.052149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.051949 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b70536d4-1f3e-456d-9c5c-63f25de2a77f" volumeName="kubernetes.io/projected/b70536d4-1f3e-456d-9c5c-63f25de2a77f-kube-api-access-fztvl" seLinuxMountContext="" Nov 03 03:01:05.052149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.051966 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c02e7385-46cc-46a6-822f-f47fe2819a3f" volumeName="kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-default-certificate" seLinuxMountContext="" Nov 03 03:01:05.052149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.052002 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fef674c6-31e4-4f47-b723-19bc2b38502c" volumeName="kubernetes.io/projected/fef674c6-31e4-4f47-b723-19bc2b38502c-kube-api-access-79gl9" seLinuxMountContext="" Nov 03 03:01:05.052149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.052033 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa09e76-0e81-4449-a649-1b5705b8d5bd" volumeName="kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-snapshots" seLinuxMountContext="" Nov 03 03:01:05.052149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.052074 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc36c84-f2e0-4f30-b74e-73d169c91ca8" volumeName="kubernetes.io/projected/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-kube-api-access-777xw" seLinuxMountContext="" Nov 03 03:01:05.052149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.052103 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8aed8e-c303-437e-9e6e-37d9334a1541" volumeName="kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s" seLinuxMountContext="" Nov 03 03:01:05.052149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.052129 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa09e76-0e81-4449-a649-1b5705b8d5bd" volumeName="kubernetes.io/projected/8fa09e76-0e81-4449-a649-1b5705b8d5bd-kube-api-access-qtspf" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.052299 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc36c84-f2e0-4f30-b74e-73d169c91ca8" volumeName="kubernetes.io/secret/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-cert" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053102 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59642d89-3475-44f2-8ddd-c95da8e30f44" volumeName="kubernetes.io/secret/59642d89-3475-44f2-8ddd-c95da8e30f44-serving-cert" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053137 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af" volumeName="kubernetes.io/secret/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053154 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72608a5a-a7a0-43e7-a9c0-bc681d988262" volumeName="kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053168 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72608a5a-a7a0-43e7-a9c0-bc681d988262" volumeName="kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053192 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa09e76-0e81-4449-a649-1b5705b8d5bd" volumeName="kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-tmp" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053205 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" volumeName="kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053218 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c02e7385-46cc-46a6-822f-f47fe2819a3f" volumeName="kubernetes.io/projected/c02e7385-46cc-46a6-822f-f47fe2819a3f-kube-api-access-gmg4d" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053231 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e4b8195-7644-4736-af66-d32d0d48551e" volumeName="kubernetes.io/projected/0e4b8195-7644-4736-af66-d32d0d48551e-kube-api-access-42lhw" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053245 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59642d89-3475-44f2-8ddd-c95da8e30f44" volumeName="kubernetes.io/projected/59642d89-3475-44f2-8ddd-c95da8e30f44-kube-api-access-h4j48" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053258 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7adb9e-2bb1-47f2-a364-04d5ec990c3a" volumeName="kubernetes.io/secret/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-serving-cert" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053272 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fef674c6-31e4-4f47-b723-19bc2b38502c" volumeName="kubernetes.io/configmap/fef674c6-31e4-4f47-b723-19bc2b38502c-telemetry-config" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053286 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af" volumeName="kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-env-overrides" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053301 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9b9e6bc-874c-4111-adfc-f3c1d50f750a" volumeName="kubernetes.io/projected/e9b9e6bc-874c-4111-adfc-f3c1d50f750a-kube-api-access-8pxw5" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053316 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b7b1fa-14b8-47ee-a3ce-dd236209b96b" volumeName="kubernetes.io/projected/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-kube-api-access-ktp62" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053332 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eb3b873-057e-44cd-9ec6-f22dd6f809b2" volumeName="kubernetes.io/projected/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-kube-api-access-6gx4h" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053350 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96" volumeName="kubernetes.io/projected/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96-kube-api-access-ljqtl" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053368 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724f0110-6fa5-473c-a88b-c514952a50a3" volumeName="kubernetes.io/secret/724f0110-6fa5-473c-a88b-c514952a50a3-agent-certs" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053399 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58a099e-243b-4581-bebc-8ceaf23509c4" volumeName="kubernetes.io/projected/f58a099e-243b-4581-bebc-8ceaf23509c4-kube-api-access-c4gr2" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053415 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb" volumeName="kubernetes.io/projected/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-kube-api-access-sdcp5" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053431 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="297e0916-b3bb-4002-810c-390a724cd782" volumeName="kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-config" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053480 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c75a11d-6c39-4d24-ac43-0b0dc04f4faf" volumeName="kubernetes.io/projected/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-kube-api-access-hmswh" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053496 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="517134a0-0ad0-46d8-a99b-ed28cce40ae5" volumeName="kubernetes.io/configmap/517134a0-0ad0-46d8-a99b-ed28cce40ae5-serviceca" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053523 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7adb9e-2bb1-47f2-a364-04d5ec990c3a" volumeName="kubernetes.io/configmap/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-config" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053535 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83b404fd-1b9b-4088-ba59-d9997f5a1558" volumeName="kubernetes.io/projected/83b404fd-1b9b-4088-ba59-d9997f5a1558-kube-api-access-crsn5" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053549 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ae62be-5346-4aff-b350-305a9a5e1a6d" volumeName="kubernetes.io/projected/50ae62be-5346-4aff-b350-305a9a5e1a6d-kube-api-access-2g5w4" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053563 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59642d89-3475-44f2-8ddd-c95da8e30f44" volumeName="kubernetes.io/configmap/59642d89-3475-44f2-8ddd-c95da8e30f44-config" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053576 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72608a5a-a7a0-43e7-a9c0-bc681d988262" volumeName="kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053588 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72608a5a-a7a0-43e7-a9c0-bc681d988262" volumeName="kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053601 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72608a5a-a7a0-43e7-a9c0-bc681d988262" volumeName="kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053614 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" volumeName="kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053626 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" volumeName="kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053639 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb498359-0464-4428-8a7d-4901cea017ac" volumeName="kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-tmp" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053650 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72608a5a-a7a0-43e7-a9c0-bc681d988262" volumeName="kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053658 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b7b1fa-14b8-47ee-a3ce-dd236209b96b" volumeName="kubernetes.io/configmap/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-config-volume" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053667 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b7b1fa-14b8-47ee-a3ce-dd236209b96b" volumeName="kubernetes.io/empty-dir/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-tmp-dir" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053674 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1714a90e-d9bd-4a76-bc61-295f6978bf17" volumeName="kubernetes.io/projected/1714a90e-d9bd-4a76-bc61-295f6978bf17-kube-api-access-d2xgb" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053681 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2469c20c-30a7-41bb-b726-d62d9324f693" volumeName="kubernetes.io/configmap/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-config" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053690 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="297e0916-b3bb-4002-810c-390a724cd782" volumeName="kubernetes.io/projected/297e0916-b3bb-4002-810c-390a724cd782-kube-api-access-lf5fl" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053699 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72608a5a-a7a0-43e7-a9c0-bc681d988262" volumeName="kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053705 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa09e76-0e81-4449-a649-1b5705b8d5bd" volumeName="kubernetes.io/secret/8fa09e76-0e81-4449-a649-1b5705b8d5bd-serving-cert" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053714 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" volumeName="kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053721 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b70536d4-1f3e-456d-9c5c-63f25de2a77f" volumeName="kubernetes.io/empty-dir/b70536d4-1f3e-456d-9c5c-63f25de2a77f-tmp-dir" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053728 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb" volumeName="kubernetes.io/configmap/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-cabundle" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053736 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f60386-016f-4213-ad61-462deca38b28" volumeName="kubernetes.io/projected/19f60386-016f-4213-ad61-462deca38b28-kube-api-access-stnf6" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053744 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2469c20c-30a7-41bb-b726-d62d9324f693" volumeName="kubernetes.io/projected/2469c20c-30a7-41bb-b726-d62d9324f693-kube-api-access-gw5tr" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053752 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c75a11d-6c39-4d24-ac43-0b0dc04f4faf" volumeName="kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-daemon-config" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053762 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57dbddca-a044-4478-8cfe-9867494e4291" volumeName="kubernetes.io/projected/57dbddca-a044-4478-8cfe-9867494e4291-kube-api-access-dsgnt" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053769 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68969cec-a620-4bd5-bfd1-cd203d0822b8" volumeName="kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053777 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="724f0110-6fa5-473c-a88b-c514952a50a3" volumeName="kubernetes.io/configmap/724f0110-6fa5-473c-a88b-c514952a50a3-konnectivity-ca" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053785 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09f7a7ee-de2c-4c6d-8ed8-39d5405848e2" volumeName="kubernetes.io/projected/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2-kube-api-access-f892q" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053793 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" volumeName="kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053800 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" volumeName="kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053808 5117 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb498359-0464-4428-8a7d-4901cea017ac" volumeName="kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-etc-tuned" seLinuxMountContext="" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053816 5117 reconstruct.go:97] "Volume reconstruction finished" Nov 03 03:01:05.056477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.053831 5117 reconciler.go:29] "Reconciler: start to sync state" Nov 03 03:01:05.064570 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.064529 5117 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 03:01:05.071156 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.071131 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/2.log" Nov 03 03:01:05.071235 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.071131 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/2.log" Nov 03 03:01:05.081829 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.081807 5117 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 03:01:05.081829 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.081829 5117 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 03:01:05.081983 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.081846 5117 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 03:01:05.081983 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.081883 5117 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 03:01:05.103200 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.103169 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/2.log" Nov 03 03:01:05.103356 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.103207 5117 generic.go:356] "Generic (PLEG): container finished" podID="297e0916-b3bb-4002-810c-390a724cd782" containerID="e08d108d1e4c3e865addb1011469ffd6b7439dd514415798897fcd17f9adbaf6" exitCode=255 Nov 03 03:01:05.107636 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.107609 5117 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="ade6fc237b1fc4f1f6b2f031aff989babf7dab808a6185d6c6adee6c741f26d3" exitCode=0 Nov 03 03:01:05.107636 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.107632 5117 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="de3e021eb64ee71d17ddf8dde8e12d8d6ae68cf3b974ce4d928a2a0f7c557df0" exitCode=0 Nov 03 03:01:05.107636 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.107638 5117 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="ea10e11ee1f9a4a429cf6cb5a7a6c4c80977e674e9676913f3def78022aac3d8" exitCode=0 Nov 03 03:01:05.107832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.107644 5117 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="c782f0535de8a568c100c6583b5a460eba8a59d966599fc3c4fd6a510bbf09e7" exitCode=0 Nov 03 03:01:05.107832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.107650 5117 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="2b47f5b7bd7e58783d6dbc9a954d10edf2d1a072e9395571db372c9bb1bb36ae" exitCode=0 Nov 03 03:01:05.107832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.107655 5117 generic.go:356] "Generic (PLEG): container finished" podID="57dbddca-a044-4478-8cfe-9867494e4291" containerID="6a8cc15861287e2bc932f28c40da9c9d972ed37cf717f39668aca59db4cb8fb2" exitCode=0 Nov 03 03:01:05.125266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.125223 5117 generic.go:356] "Generic (PLEG): container finished" podID="e9fdaf3bc04b642a1b5c4542f0b4daec" containerID="7abcaa7899297e64a6b03d70e15ea464ba9f313c50caaeaabf859fae0c58c94e" exitCode=0 Nov 03 03:01:05.144585 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.144553 5117 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:01:05.152256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.152234 5117 manager.go:324] Recovery completed Nov 03 03:01:05.170892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.170821 5117 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:05.172357 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.172337 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:05.172406 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.172376 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:05.172406 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.172387 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:05.175138 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175123 5117 cpu_manager.go:226] "Starting" policy="none" Nov 03 03:01:05.175177 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175137 5117 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 03:01:05.175177 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175149 5117 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:05.175252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175155 5117 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 03:01:05.175405 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175391 5117 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 03:01:05.175436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175406 5117 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 03:01:05.175436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175424 5117 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 03:01:05.175436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175430 5117 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 03:01:05.175436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175435 5117 policy_none.go:50] "Start" Nov 03 03:01:05.175555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175492 5117 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 03:01:05.175555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175504 5117 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 03:01:05.175640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175630 5117 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 03:01:05.175669 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175641 5117 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 03:01:05.175669 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.175647 5117 policy_none.go:44] "Start" Nov 03 03:01:05.176897 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.176869 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:05.177027 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.176908 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:05.177027 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.176924 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:05.181799 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.181768 5117 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 03:01:05.181959 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.181938 5117 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 03:01:05.184371 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184359 5117 manager.go:341] "Starting Device Plugin manager" Nov 03 03:01:05.184418 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.184390 5117 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 03:01:05.184418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184399 5117 server.go:87] "Starting device plugin registration server" Nov 03 03:01:05.184607 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184596 5117 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 03:01:05.184636 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184610 5117 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 03:01:05.184726 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184708 5117 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 03:01:05.184901 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184862 5117 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 03:01:05.185029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184909 5117 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 03:01:05.185029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184917 5117 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 03:01:05.185029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.184948 5117 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T03:01:05.184897708Z","UUID":"f4de4fbf-bc53-4e78-84f8-379ce215d320","Handler":null,"Name":"","Endpoint":""} Nov 03 03:01:05.185967 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.185947 5117 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 03:01:05.186035 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.185974 5117 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 03:01:05.186634 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.186611 5117 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 03:01:05.186737 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.186656 5117 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-120.us-west-1.compute.internal\" not found" Nov 03 03:01:05.249892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.249856 5117 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 03:01:05.250075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.249970 5117 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:05.250075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.249985 5117 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:05.284996 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.284961 5117 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 03:01:05.286289 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.286265 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 03:01:05.286420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.286297 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 03:01:05.286420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.286308 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 03:01:05.286420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.286358 5117 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.299286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.299262 5117 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:05.307198 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.307176 5117 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 03:01:05.382162 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.382112 5117 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal"] Nov 03 03:01:05.382966 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.382946 5117 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.383075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.382954 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"e9fdaf3bc04b642a1b5c4542f0b4daec","Type":"ContainerStarted","Data":"7138db1a7786853dd4cd7a0c491c538ce539589d8451598d5d0e97987106184b"} Nov 03 03:01:05.383075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.382999 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"e9fdaf3bc04b642a1b5c4542f0b4daec","Type":"ContainerDied","Data":"7abcaa7899297e64a6b03d70e15ea464ba9f313c50caaeaabf859fae0c58c94e"} Nov 03 03:01:05.383075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.383013 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"e9fdaf3bc04b642a1b5c4542f0b4daec","Type":"ContainerStarted","Data":"7dcc5bf9bc589a4023b99c9a6191702ad99e8f41a3ffc83eba5370cd791acad2"} Nov 03 03:01:05.383075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.383039 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"143abffde7987a79bc89f8333af31943","Type":"ContainerStarted","Data":"b7a1dcf474e5d04e386d0f66bf30e2920d4c8005e8bfca678f31f3c46fedbe0f"} Nov 03 03:01:05.383075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.383050 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" event={"ID":"143abffde7987a79bc89f8333af31943","Type":"ContainerStarted","Data":"6a98912186ec65b128bc56e3cba4ee633c4383023b726603748559b9a439c222"} Nov 03 03:01:05.383232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.383181 5117 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.403148 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.403124 5117 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.403237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.403193 5117 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.403839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.403813 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeNotReady" Nov 03 03:01:05.404007 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.403852 5117 setters.go:546] "Node became not ready" node="ip-10-0-138-120.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T03:01:05Z","lastTransitionTime":"2026-11-03T03:01:05Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 03:01:05.435491 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.435403 5117 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-120.us-west-1.compute.internal" is owned by different node (oldNodeID="8f9585d6-3b6f-4815-88a6-2900b9d03d3b", newNodeID=""), cleaning up... Nov 03 03:01:05.448028 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.448003 5117 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:05.448138 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.448064 5117 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.448960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.448946 5117 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:05.449049 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:05.448992 5117 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.451112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.451090 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.451173 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.451121 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.451173 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.451140 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/143abffde7987a79bc89f8333af31943-config\") pod \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"143abffde7987a79bc89f8333af31943\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.551471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.551415 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.551471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.551361 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.551714 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.551487 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.551714 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.551504 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/143abffde7987a79bc89f8333af31943-config\") pod \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"143abffde7987a79bc89f8333af31943\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.551714 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.551531 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/143abffde7987a79bc89f8333af31943-config\") pod \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"143abffde7987a79bc89f8333af31943\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.551714 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.551590 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9fdaf3bc04b642a1b5c4542f0b4daec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" (UID: \"e9fdaf3bc04b642a1b5c4542f0b4daec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:05.675108 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.675079 5117 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-120.us-west-1.compute.internal" event="NodeReady" Nov 03 03:01:05.675281 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:05.675186 5117 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 03:01:06.031821 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.031791 5117 apiserver.go:52] "Watching apiserver" Nov 03 03:01:06.150388 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.150357 5117 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:06.150585 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.150501 5117 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:06.188925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.188885 5117 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 03:01:06.189735 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.189701 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xw724","openshift-multus/multus-additional-cni-plugins-hzvq6","kube-system/konnectivity-agent-qg9cd","openshift-image-registry/node-ca-vnrpn","openshift-dns/node-resolver-dtzkv","openshift-network-diagnostics/network-check-target-ldf8b","openshift-ingress-canary/ingress-canary-lctfx","openshift-cluster-node-tuning-operator/tuned-pmq44","openshift-network-operator/iptables-alerter-2fd9d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq","openshift-image-registry/image-registry-5bf7f94cf9-nvxkf","kube-system/kas-connection-checker-685895c7c9-d8k4b","openshift-ingress/router-default-c5c7c7bf7-x5tt5","openshift-console-operator/console-operator-7cd56b8dbc-f869c","openshift-network-console/networking-console-plugin-86c478bb88-bhczt","openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4","openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g","openshift-insights/insights-runtime-extractor-762ml","openshift-insights/periodic-gathering-pr56b-bzv26","kube-system/global-pull-secret-syncer-j76qm","kube-system/kas-connection-checker-685895c7c9-5r574","openshift-insights/insights-operator-7c4986d449-2hdcl","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr","openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr","openshift-service-ca/service-ca-5fffc97bdf-4lnbs","openshift-image-registry/image-registry-675ff945d6-2qfxd","openshift-multus/multus-ssswp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f","openshift-ovn-kubernetes/ovnkube-node-7glvk","openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm","kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal","kube-system/kas-connection-checker-685895c7c9-nz8pk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l","openshift-dns/dns-default-dwkvw"] Nov 03 03:01:06.191010 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.190986 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.192076 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.192044 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.192470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.192427 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.193079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.193061 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.194606 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.194586 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:06.200353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.200323 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:06.204315 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.204294 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.205232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.205215 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.205838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.205819 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.214036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.214005 5117 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:06.214187 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.214082 5117 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:06.214722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.214706 5117 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 03:01:06.214777 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.214750 5117 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-120.us-west-1.compute.internal" Nov 03 03:01:06.246960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.246927 5117 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 03:01:06.255300 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255267 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tl5c\" (UniqueName: \"kubernetes.io/projected/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-kube-api-access-5tl5c\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:01:06.255422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255317 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysconfig\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.255422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255347 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.255422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255372 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-kubelet\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.255639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255492 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-sys-fs\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.255639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255527 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.255639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255577 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j48\" (UniqueName: \"kubernetes.io/projected/59642d89-3475-44f2-8ddd-c95da8e30f44-kube-api-access-h4j48\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:01:06.255746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255653 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-slash\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.255746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255708 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.255808 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255743 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.255885 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255866 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.255923 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255904 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-bin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.255957 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255942 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-multus\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.255988 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255963 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-cabundle\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:06.256024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.255986 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.256063 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256026 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.256106 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256067 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-run\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.256143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256126 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-etc-tuned\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.256183 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256171 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-env-overrides\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.256217 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256192 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-sys\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.256217 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256208 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-host\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.256217 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256209 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-etc-tuned\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.256305 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256234 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq6x\" (UniqueName: \"kubernetes.io/projected/fb498359-0464-4428-8a7d-4901cea017ac-kube-api-access-tgq6x\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.256353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256338 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-cnibin\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.256386 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256376 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-registration-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.256431 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256418 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5w4\" (UniqueName: \"kubernetes.io/projected/50ae62be-5346-4aff-b350-305a9a5e1a6d-kube-api-access-2g5w4\") pod \"kas-connection-checker-685895c7c9-5r574\" (UID: \"50ae62be-5346-4aff-b350-305a9a5e1a6d\") " pod="kube-system/kas-connection-checker-685895c7c9-5r574" Nov 03 03:01:06.256492 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256468 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-default-certificate\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.256537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256502 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.256537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256526 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-iptables-alerter-script\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:01:06.256637 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256618 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.256675 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256623 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.256711 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256675 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lhw\" (UniqueName: \"kubernetes.io/projected/0e4b8195-7644-4736-af66-d32d0d48551e-kube-api-access-42lhw\") pod \"kas-connection-checker-685895c7c9-d8k4b\" (UID: \"0e4b8195-7644-4736-af66-d32d0d48551e\") " pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" Nov 03 03:01:06.256742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256704 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:01:06.256742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256733 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-systemd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.256800 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256759 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-bin\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.256800 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256789 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pqmg\" (UniqueName: \"kubernetes.io/projected/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-kube-api-access-9pqmg\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.256866 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256828 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.256986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.256962 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztvl\" (UniqueName: \"kubernetes.io/projected/b70536d4-1f3e-456d-9c5c-63f25de2a77f-kube-api-access-fztvl\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:01:06.257019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257006 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtspf\" (UniqueName: \"kubernetes.io/projected/8fa09e76-0e81-4449-a649-1b5705b8d5bd-kube-api-access-qtspf\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.257050 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257032 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59642d89-3475-44f2-8ddd-c95da8e30f44-serving-cert\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:01:06.257082 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257061 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9644b939-f36f-49ec-8388-c0e2c5bdce5a-data-volume\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.257185 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257085 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrcw\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.257185 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257162 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-hostroot\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.257275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257193 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcp5\" (UniqueName: \"kubernetes.io/projected/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-kube-api-access-sdcp5\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:06.257275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257193 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9644b939-f36f-49ec-8388-c0e2c5bdce5a-data-volume\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.257275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257217 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-host-slash\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:01:06.257275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257239 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-config\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.257414 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257390 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.257478 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257435 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59642d89-3475-44f2-8ddd-c95da8e30f44-config\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:01:06.257557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257502 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-system-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.257557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257539 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8lc\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.257646 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257568 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5fl\" (UniqueName: \"kubernetes.io/projected/297e0916-b3bb-4002-810c-390a724cd782-kube-api-access-lf5fl\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.257740 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257723 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-lib-modules\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.257807 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257751 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktp62\" (UniqueName: \"kubernetes.io/projected/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-kube-api-access-ktp62\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.257807 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257792 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gr2\" (UniqueName: \"kubernetes.io/projected/f58a099e-243b-4581-bebc-8ceaf23509c4-kube-api-access-c4gr2\") pod \"network-check-source-6f78d59f99-tjzsm\" (UID: \"f58a099e-243b-4581-bebc-8ceaf23509c4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" Nov 03 03:01:06.257916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257836 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-node-log\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.257916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257855 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-tmp\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.257916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257877 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b70536d4-1f3e-456d-9c5c-63f25de2a77f-hosts-file\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:01:06.258040 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257936 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:06.258040 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257954 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn5q2\" (UniqueName: \"kubernetes.io/projected/517134a0-0ad0-46d8-a99b-ed28cce40ae5-kube-api-access-dn5q2\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:01:06.258040 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257970 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnf6\" (UniqueName: \"kubernetes.io/projected/19f60386-016f-4213-ad61-462deca38b28-kube-api-access-stnf6\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.258040 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.257974 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb498359-0464-4428-8a7d-4901cea017ac-tmp\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.258418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258001 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-snapshots\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.258496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258438 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.258496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258481 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-service-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.258619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258511 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxw5\" (UniqueName: \"kubernetes.io/projected/e9b9e6bc-874c-4111-adfc-f3c1d50f750a-kube-api-access-8pxw5\") pod \"migrator-66df5d496b-gcxsr\" (UID: \"e9b9e6bc-874c-4111-adfc-f3c1d50f750a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" Nov 03 03:01:06.258619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258538 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/297e0916-b3bb-4002-810c-390a724cd782-serving-cert\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.258619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258564 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-config\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.258619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258589 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/517134a0-0ad0-46d8-a99b-ed28cce40ae5-serviceca\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:01:06.258619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258615 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70536d4-1f3e-456d-9c5c-63f25de2a77f-tmp-dir\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258621 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-snapshots\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258644 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cni-binary-copy\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258673 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258703 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258722 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70536d4-1f3e-456d-9c5c-63f25de2a77f-tmp-dir\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258735 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/724f0110-6fa5-473c-a88b-c514952a50a3-konnectivity-ca\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258784 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovn-node-metrics-cert\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258816 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-kubelet-config\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.258862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258845 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-etc-kubernetes\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258880 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa09e76-0e81-4449-a649-1b5705b8d5bd-serving-cert\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258912 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/517134a0-0ad0-46d8-a99b-ed28cce40ae5-host\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.258985 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5tr\" (UniqueName: \"kubernetes.io/projected/2469c20c-30a7-41bb-b726-d62d9324f693-kube-api-access-gw5tr\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259016 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-script-lib\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259043 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gk4\" (UniqueName: \"kubernetes.io/projected/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-api-access-h6gk4\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259069 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259097 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fef674c6-31e4-4f47-b723-19bc2b38502c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.259168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259134 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259170 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgnt\" (UniqueName: \"kubernetes.io/projected/57dbddca-a044-4478-8cfe-9867494e4291-kube-api-access-dsgnt\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259219 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-777xw\" (UniqueName: \"kubernetes.io/projected/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-kube-api-access-777xw\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259301 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmg4d\" (UniqueName: \"kubernetes.io/projected/c02e7385-46cc-46a6-822f-f47fe2819a3f-kube-api-access-gmg4d\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259363 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259394 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gx4h\" (UniqueName: \"kubernetes.io/projected/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-kube-api-access-6gx4h\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259437 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-netns\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259488 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-socket-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.259544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259515 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gl9\" (UniqueName: \"kubernetes.io/projected/fef674c6-31e4-4f47-b723-19bc2b38502c-kube-api-access-79gl9\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.259836 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259555 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/724f0110-6fa5-473c-a88b-c514952a50a3-agent-certs\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:01:06.260108 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.259689 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-log-socket\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.260180 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260123 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-tmp-dir\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.260180 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260147 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.260180 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260169 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f892q\" (UniqueName: \"kubernetes.io/projected/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2-kube-api-access-f892q\") pod \"volume-data-source-validator-58c9f5f46d-rgsmd\" (UID: \"09f7a7ee-de2c-4c6d-8ed8-39d5405848e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" Nov 03 03:01:06.260325 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260186 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-key\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:06.260325 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260202 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-etc-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.260325 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260302 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.260325 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260303 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-tmp-dir\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.260507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260329 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.260507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260363 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-conf-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.260507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260394 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.260507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260394 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.260507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260422 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-ovn\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.260507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260481 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260510 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260531 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260551 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-systemd\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260612 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260628 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260667 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cnibin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260725 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.260773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260726 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmswh\" (UniqueName: \"kubernetes.io/projected/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-kube-api-access-hmswh\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.261070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260849 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xgb\" (UniqueName: \"kubernetes.io/projected/1714a90e-d9bd-4a76-bc61-295f6978bf17-kube-api-access-d2xgb\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:06.261070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260937 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-var-lib-kubelet\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.261070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260965 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.261070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.260998 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-device-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.261190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261080 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-daemon-config\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.261190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261112 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.261190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261142 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.261190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261181 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.261304 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261210 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-netns\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.261304 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261287 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-multus-certs\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.261364 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261305 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-tmp\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.261396 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261338 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-system-cni-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.261396 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261382 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa09e76-0e81-4449-a649-1b5705b8d5bd-tmp\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.261476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261392 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:06.261476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261437 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-kubelet\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.261539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261488 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-systemd-units\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.261539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261504 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-var-lib-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.261539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261519 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9644b939-f36f-49ec-8388-c0e2c5bdce5a-crio-socket\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.261627 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261540 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.261627 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261605 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-modprobe-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.261686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261629 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:06.261686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261668 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.261744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261698 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-config-volume\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.261744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261739 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.261831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261818 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-trusted-ca\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.261895 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261856 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:01:06.261972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261959 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.262006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.261989 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-socket-dir-parent\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.262038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262019 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsn5\" (UniqueName: \"kubernetes.io/projected/83b404fd-1b9b-4088-ba59-d9997f5a1558-kube-api-access-crsn5\") pod \"kas-connection-checker-685895c7c9-nz8pk\" (UID: \"83b404fd-1b9b-4088-ba59-d9997f5a1558\") " pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" Nov 03 03:01:06.262067 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262036 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.262067 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262058 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-dbus\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.262210 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262196 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.262266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262252 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-cert\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:01:06.262370 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262357 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-conf\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.262402 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262394 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hr7\" (UniqueName: \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.262501 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262487 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.262578 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262561 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-stats-auth\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.262649 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262628 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-netd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.262682 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262668 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-kubernetes\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.262713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262702 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljqtl\" (UniqueName: \"kubernetes.io/projected/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96-kube-api-access-ljqtl\") pod \"migrator-66df5d496b-cgn4g\" (UID: \"6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" Nov 03 03:01:06.262744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262732 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-k8s-cni-cncf-io\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.262858 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262847 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.262904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.262887 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.268234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.268216 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 03:01:06.268376 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.268354 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.269734 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.269717 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/297e0916-b3bb-4002-810c-390a724cd782-serving-cert\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.271103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271089 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.271182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271104 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 03:01:06.271182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271132 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 03:01:06.271182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271174 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.271379 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271366 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.271547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271535 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.271660 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271647 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 03:01:06.271716 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271680 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 03:01:06.271795 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271784 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 03:01:06.271840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.271825 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.274260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274117 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 03:01:06.274260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274131 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.274260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274138 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.274489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274266 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.274489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274350 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.274489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274408 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.274489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274469 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.274686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274521 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.274686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274597 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.274686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274673 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 03:01:06.274686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274681 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 03:01:06.274966 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274784 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 03:01:06.274966 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.274793 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.277147 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.277127 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-default-certificate\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.277264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.277248 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59642d89-3475-44f2-8ddd-c95da8e30f44-serving-cert\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:01:06.278124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278107 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/59642d89-3475-44f2-8ddd-c95da8e30f44-config\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:01:06.278603 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278586 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.278663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278617 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-tbkxl\"" Nov 03 03:01:06.278663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278627 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 03:01:06.278741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278658 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rnqls\"" Nov 03 03:01:06.278741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278648 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.278741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278689 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.278838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278789 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 03:01:06.278838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278825 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.279020 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.278998 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 03:01:06.279190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279170 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovn-node-metrics-cert\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.279321 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279302 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fa09e76-0e81-4449-a649-1b5705b8d5bd-serving-cert\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.279397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279362 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.279397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279378 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 03:01:06.279548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279382 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 03:01:06.279548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279504 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.279548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279517 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.279548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279517 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.279741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279555 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 03:01:06.279741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279597 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.279741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279635 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 03:01:06.279741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.279638 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 03:01:06.280056 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.280027 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 03:01:06.280323 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.280301 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.280843 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.280817 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.281029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.281012 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:01:06.284435 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.284320 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-stats-auth\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.284435 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.284374 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.284788 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.284770 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:01:06.286615 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.286601 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-env-overrides\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.288942 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.288922 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-service-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.289003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.288929 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-config\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.321522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.321491 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 03:01:06.322129 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.322107 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 03:01:06.327969 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.327935 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-config\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.329623 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.329599 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fef674c6-31e4-4f47-b723-19bc2b38502c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.330141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330112 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 03:01:06.330249 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330187 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 03:01:06.330308 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330250 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 03:01:06.330308 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330304 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 03:01:06.330412 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330343 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 03:01:06.330412 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330358 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 03:01:06.330412 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330358 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 03:01:06.330412 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330369 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 03:01:06.330838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.330819 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.331074 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331058 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 03:01:06.331161 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331144 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.331376 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331358 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-daemon-config\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.331436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331365 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 03:01:06.331570 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331553 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.331635 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331591 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.331685 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331669 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.331685 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331671 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 03:01:06.331908 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331845 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.331908 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331861 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.332036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331917 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.332036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331953 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.332036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.331983 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-config-volume\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.332204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.332042 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.332204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.332121 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.332204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.332126 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.332204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.332189 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.332387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.332271 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297e0916-b3bb-4002-810c-390a724cd782-trusted-ca\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.332460 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.332420 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.333017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333001 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 03:01:06.333214 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333185 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 03:01:06.333214 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333188 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 03:01:06.333393 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333262 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.333527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333503 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 03:01:06.333527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333510 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.333670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333602 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.333670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333638 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 03:01:06.333670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333640 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.333776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333679 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 03:01:06.333776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333710 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.333776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.333688 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 03:01:06.334080 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.334064 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tl5c\" (UniqueName: \"kubernetes.io/projected/6b7adb9e-2bb1-47f2-a364-04d5ec990c3a-kube-api-access-5tl5c\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gph9f\" (UID: \"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" Nov 03 03:01:06.336014 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.335999 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.336187 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.336169 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.336222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.336182 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-cabundle\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:06.337189 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.337163 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.337189 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.337177 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.337345 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.337253 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-iptables-alerter-script\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:01:06.337689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.337670 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq6x\" (UniqueName: \"kubernetes.io/projected/fb498359-0464-4428-8a7d-4901cea017ac-kube-api-access-tgq6x\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.338423 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.338405 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-nginx-conf\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:06.339188 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.339162 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/517134a0-0ad0-46d8-a99b-ed28cce40ae5-serviceca\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:01:06.339275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.339165 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cni-binary-copy\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.339275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.339196 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/724f0110-6fa5-473c-a88b-c514952a50a3-konnectivity-ca\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:01:06.339275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.339257 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa09e76-0e81-4449-a649-1b5705b8d5bd-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.339438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.339290 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-ovnkube-script-lib\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.339438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.339434 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5w4\" (UniqueName: \"kubernetes.io/projected/50ae62be-5346-4aff-b350-305a9a5e1a6d-kube-api-access-2g5w4\") pod \"kas-connection-checker-685895c7c9-5r574\" (UID: \"50ae62be-5346-4aff-b350-305a9a5e1a6d\") " pod="kube-system/kas-connection-checker-685895c7c9-5r574" Nov 03 03:01:06.341401 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.341383 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.342511 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.342494 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/57dbddca-a044-4478-8cfe-9867494e4291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.346610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.346592 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4j48\" (UniqueName: \"kubernetes.io/projected/59642d89-3475-44f2-8ddd-c95da8e30f44-kube-api-access-h4j48\") pod \"service-ca-operator-544d749c8c-gxsq4\" (UID: \"59642d89-3475-44f2-8ddd-c95da8e30f44\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" Nov 03 03:01:06.348736 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.348718 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 03:01:06.354673 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.354643 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pqmg\" (UniqueName: \"kubernetes.io/projected/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-kube-api-access-9pqmg\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.354840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.354694 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn5q2\" (UniqueName: \"kubernetes.io/projected/517134a0-0ad0-46d8-a99b-ed28cce40ae5-kube-api-access-dn5q2\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:01:06.354926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.354905 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.355246 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.355228 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrcw\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.355319 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.355277 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5fl\" (UniqueName: \"kubernetes.io/projected/297e0916-b3bb-4002-810c-390a724cd782-kube-api-access-lf5fl\") pod \"console-operator-7cd56b8dbc-f869c\" (UID: \"297e0916-b3bb-4002-810c-390a724cd782\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.356222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.356193 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xgb\" (UniqueName: \"kubernetes.io/projected/1714a90e-d9bd-4a76-bc61-295f6978bf17-kube-api-access-d2xgb\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:06.356438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.356422 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.356571 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.356555 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtspf\" (UniqueName: \"kubernetes.io/projected/8fa09e76-0e81-4449-a649-1b5705b8d5bd-kube-api-access-qtspf\") pod \"insights-operator-7c4986d449-2hdcl\" (UID: \"8fa09e76-0e81-4449-a649-1b5705b8d5bd\") " pod="openshift-insights/insights-operator-7c4986d449-2hdcl" Nov 03 03:01:06.359238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.359222 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4k7s\" (UniqueName: \"kubernetes.io/projected/4b8aed8e-c303-437e-9e6e-37d9334a1541-kube-api-access-r4k7s\") pod \"network-check-target-ldf8b\" (UID: \"4b8aed8e-c303-437e-9e6e-37d9334a1541\") " pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:01:06.363372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363352 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-conf-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.363497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363385 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-ovn\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.363497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363409 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-systemd\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.363497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363433 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.363497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363489 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-conf-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363503 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-ovn\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363548 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363507 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cnibin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363560 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-cnibin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363518 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-systemd\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363586 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-var-lib-kubelet\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363623 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363651 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-var-lib-kubelet\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363668 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-device-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363700 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.363713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363716 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-device-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363727 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-netns\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363749 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363754 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-multus-certs\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363807 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-netns\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363810 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-multus-certs\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363839 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-system-cni-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363861 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363889 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-kubelet\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363905 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-system-cni-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363914 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-systemd-units\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363950 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-systemd-units\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.363953 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-kubelet\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364011 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-var-lib-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364021 5117 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364040 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9644b939-f36f-49ec-8388-c0e2c5bdce5a-crio-socket\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364074 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364077 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-var-lib-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364101 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-modprobe-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364126 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9644b939-f36f-49ec-8388-c0e2c5bdce5a-crio-socket\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364127 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364189 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364213 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-socket-dir-parent\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.364219 5117 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364226 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-modprobe-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.364222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364237 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364267 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-dbus\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364276 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-multus-socket-dir-parent\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364293 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-conf\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.364319 5117 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364323 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.364357 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.864335277 +0000 UTC m=+1.894933806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364357 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-dbus\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.364393 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.864382478 +0000 UTC m=+1.894981062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364411 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-netd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364436 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-kubernetes\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364480 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-netd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364489 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-k8s-cni-cncf-io\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364498 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364509 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-conf\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364521 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysconfig\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364550 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364554 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-kubernetes\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364554 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-run-k8s-cni-cncf-io\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364575 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-kubelet\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364600 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-sys-fs\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364626 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-slash\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364649 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364657 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-kubelet\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364669 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysconfig\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364744 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364782 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364788 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364742 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-sys-fs\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364813 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-slash\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364806 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-bin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364835 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-bin\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364855 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-multus\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364874 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364891 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-run\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364936 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-run\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364934 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-host-var-lib-cni-multus\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364953 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364947 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-sys\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.364995 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-host\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365007 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-sys\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365023 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-cnibin\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365056 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-host\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365076 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-registration-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365107 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57dbddca-a044-4478-8cfe-9867494e4291-cnibin\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365111 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-systemd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365140 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-bin\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365174 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-hostroot\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365185 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-registration-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365196 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-run-systemd\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365202 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-cni-bin\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365201 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-host-slash\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365237 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-host-slash\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365242 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-hostroot\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365275 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.365286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365320 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-system-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365338 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365350 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-lib-modules\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365377 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-node-log\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365392 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-system-cni-dir\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365402 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b70536d4-1f3e-456d-9c5c-63f25de2a77f-hosts-file\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365472 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-node-log\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365472 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b70536d4-1f3e-456d-9c5c-63f25de2a77f-hosts-file\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365502 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-lib-modules\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365493 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365546 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.365582 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.865566144 +0000 UTC m=+1.896164676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365614 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-kubelet-config\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365641 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-etc-kubernetes\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365664 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68969cec-a620-4bd5-bfd1-cd203d0822b8-kubelet-config\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365667 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/517134a0-0ad0-46d8-a99b-ed28cce40ae5-host\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365695 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365702 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-etc-kubernetes\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365724 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365751 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-netns\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365760 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/517134a0-0ad0-46d8-a99b-ed28cce40ae5-host\") pod \"node-ca-vnrpn\" (UID: \"517134a0-0ad0-46d8-a99b-ed28cce40ae5\") " pod="openshift-image-registry/node-ca-vnrpn" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365776 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-socket-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365801 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-log-socket\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365828 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-etc-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365836 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-host-run-netns\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365836 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fb498359-0464-4428-8a7d-4901cea017ac-etc-sysctl-d\") pod \"tuned-pmq44\" (UID: \"fb498359-0464-4428-8a7d-4901cea017ac\") " pod="openshift-cluster-node-tuning-operator/tuned-pmq44" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365887 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19f60386-016f-4213-ad61-462deca38b28-socket-dir\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365906 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-log-socket\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.365926 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af-etc-openvswitch\") pod \"ovnkube-node-7glvk\" (UID: \"6b47370a-1fb8-43ef-8ecb-c0ff2eeb56af\") " pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.366944 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.367067 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 03:01:06.367764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.367468 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"image-registry-5bf7f94cf9-nvxkf\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.369352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.367850 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2469c20c-30a7-41bb-b726-d62d9324f693-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.369352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.368252 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.369352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.368361 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fef674c6-31e4-4f47-b723-19bc2b38502c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.369352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369341 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vbzfk\"" Nov 03 03:01:06.369584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369566 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-srwkl\"" Nov 03 03:01:06.369584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369576 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 03:01:06.369878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369743 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrqj7\"" Nov 03 03:01:06.369878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369766 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 03:01:06.369878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369786 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ckmp9\"" Nov 03 03:01:06.369878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369802 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcpl5\"" Nov 03 03:01:06.369878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369825 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fg6rm\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369922 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369932 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqjzv\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369950 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369984 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fp5b4\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370004 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljqtl\" (UniqueName: \"kubernetes.io/projected/6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96-kube-api-access-ljqtl\") pod \"migrator-66df5d496b-cgn4g\" (UID: \"6f4b6983-cfe0-45f7-bc69-e3cb91d0ff96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cgn4g" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369992 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370013 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pw7w9\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.369997 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370063 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgzd9\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370074 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/724f0110-6fa5-473c-a88b-c514952a50a3-agent-certs\") pod \"konnectivity-agent-qg9cd\" (UID: \"724f0110-6fa5-473c-a88b-c514952a50a3\") " pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370084 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fz9pb\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370122 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8fw8c\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370123 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zgcvz\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370165 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d2r77\"" Nov 03 03:01:06.370184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370171 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 03:01:06.371001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370287 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tvk4\"" Nov 03 03:01:06.371001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370526 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-signing-key\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:06.371001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370585 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snr2z\"" Nov 03 03:01:06.371001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.370644 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 03:01:06.371417 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.371401 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68969cec-a620-4bd5-bfd1-cd203d0822b8-original-pull-secret\") pod \"global-pull-secret-syncer-j76qm\" (UID: \"68969cec-a620-4bd5-bfd1-cd203d0822b8\") " pod="kube-system/global-pull-secret-syncer-j76qm" Nov 03 03:01:06.372517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.372502 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-cert\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:01:06.374216 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.374202 5117 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:01:06.374307 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.374216 5117 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 03:01:06.374307 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.374243 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.874230998 +0000 UTC m=+1.904829515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:01:06.374307 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.374257 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:06.874250409 +0000 UTC m=+1.904848925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : secret "metrics-daemon-secret" not found Nov 03 03:01:06.374826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.374805 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gk4\" (UniqueName: \"kubernetes.io/projected/9644b939-f36f-49ec-8388-c0e2c5bdce5a-kube-api-access-h6gk4\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.374952 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.374910 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hr7\" (UniqueName: \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\") pod \"periodic-gathering-pr56b-bzv26\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:01:06.376863 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.376832 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-metrics-tls\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.382521 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.382431 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmswh\" (UniqueName: \"kubernetes.io/projected/4c75a11d-6c39-4d24-ac43-0b0dc04f4faf-kube-api-access-hmswh\") pod \"multus-ssswp\" (UID: \"4c75a11d-6c39-4d24-ac43-0b0dc04f4faf\") " pod="openshift-multus/multus-ssswp" Nov 03 03:01:06.383022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.382701 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztvl\" (UniqueName: \"kubernetes.io/projected/b70536d4-1f3e-456d-9c5c-63f25de2a77f-kube-api-access-fztvl\") pod \"node-resolver-dtzkv\" (UID: \"b70536d4-1f3e-456d-9c5c-63f25de2a77f\") " pod="openshift-dns/node-resolver-dtzkv" Nov 03 03:01:06.383022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.382701 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktp62\" (UniqueName: \"kubernetes.io/projected/f2b7b1fa-14b8-47ee-a3ce-dd236209b96b-kube-api-access-ktp62\") pod \"dns-default-dwkvw\" (UID: \"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b\") " pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.383022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.382788 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f892q\" (UniqueName: \"kubernetes.io/projected/09f7a7ee-de2c-4c6d-8ed8-39d5405848e2-kube-api-access-f892q\") pod \"volume-data-source-validator-58c9f5f46d-rgsmd\" (UID: \"09f7a7ee-de2c-4c6d-8ed8-39d5405848e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rgsmd" Nov 03 03:01:06.383022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.382874 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsn5\" (UniqueName: \"kubernetes.io/projected/83b404fd-1b9b-4088-ba59-d9997f5a1558-kube-api-access-crsn5\") pod \"kas-connection-checker-685895c7c9-nz8pk\" (UID: \"83b404fd-1b9b-4088-ba59-d9997f5a1558\") " pod="kube-system/kas-connection-checker-685895c7c9-nz8pk" Nov 03 03:01:06.383022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.382977 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcp5\" (UniqueName: \"kubernetes.io/projected/00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb-kube-api-access-sdcp5\") pod \"service-ca-5fffc97bdf-4lnbs\" (UID: \"00f2ca1a-b6e5-4a67-bb0c-e58bc8fa1feb\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4lnbs" Nov 03 03:01:06.383790 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.383770 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lhw\" (UniqueName: \"kubernetes.io/projected/0e4b8195-7644-4736-af66-d32d0d48551e-kube-api-access-42lhw\") pod \"kas-connection-checker-685895c7c9-d8k4b\" (UID: \"0e4b8195-7644-4736-af66-d32d0d48551e\") " pod="kube-system/kas-connection-checker-685895c7c9-d8k4b" Nov 03 03:01:06.383903 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.383886 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gr2\" (UniqueName: \"kubernetes.io/projected/f58a099e-243b-4581-bebc-8ceaf23509c4-kube-api-access-c4gr2\") pod \"network-check-source-6f78d59f99-tjzsm\" (UID: \"f58a099e-243b-4581-bebc-8ceaf23509c4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-tjzsm" Nov 03 03:01:06.383970 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.383956 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.384348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.384332 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgnt\" (UniqueName: \"kubernetes.io/projected/57dbddca-a044-4478-8cfe-9867494e4291-kube-api-access-dsgnt\") pod \"multus-additional-cni-plugins-hzvq6\" (UID: \"57dbddca-a044-4478-8cfe-9867494e4291\") " pod="openshift-multus/multus-additional-cni-plugins-hzvq6" Nov 03 03:01:06.384436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.384420 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gx4h\" (UniqueName: \"kubernetes.io/projected/0eb3b873-057e-44cd-9ec6-f22dd6f809b2-kube-api-access-6gx4h\") pod \"iptables-alerter-2fd9d\" (UID: \"0eb3b873-057e-44cd-9ec6-f22dd6f809b2\") " pod="openshift-network-operator/iptables-alerter-2fd9d" Nov 03 03:01:06.384614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.384599 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5tr\" (UniqueName: \"kubernetes.io/projected/2469c20c-30a7-41bb-b726-d62d9324f693-kube-api-access-gw5tr\") pod \"cluster-samples-operator-6ddb6c8b94-6t2pr\" (UID: \"2469c20c-30a7-41bb-b726-d62d9324f693\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.384893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.384879 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnf6\" (UniqueName: \"kubernetes.io/projected/19f60386-016f-4213-ad61-462deca38b28-kube-api-access-stnf6\") pod \"aws-ebs-csi-driver-node-svg4l\" (UID: \"19f60386-016f-4213-ad61-462deca38b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-svg4l" Nov 03 03:01:06.385042 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.385026 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmg4d\" (UniqueName: \"kubernetes.io/projected/c02e7385-46cc-46a6-822f-f47fe2819a3f-kube-api-access-gmg4d\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.386946 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.386929 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxw5\" (UniqueName: \"kubernetes.io/projected/e9b9e6bc-874c-4111-adfc-f3c1d50f750a-kube-api-access-8pxw5\") pod \"migrator-66df5d496b-gcxsr\" (UID: \"e9b9e6bc-874c-4111-adfc-f3c1d50f750a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gcxsr" Nov 03 03:01:06.389607 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.389591 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8lc\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\") pod \"image-registry-675ff945d6-2qfxd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.390547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.390531 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-777xw\" (UniqueName: \"kubernetes.io/projected/3fc36c84-f2e0-4f30-b74e-73d169c91ca8-kube-api-access-777xw\") pod \"ingress-canary-lctfx\" (UID: \"3fc36c84-f2e0-4f30-b74e-73d169c91ca8\") " pod="openshift-ingress-canary/ingress-canary-lctfx" Nov 03 03:01:06.394663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.394647 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gl9\" (UniqueName: \"kubernetes.io/projected/fef674c6-31e4-4f47-b723-19bc2b38502c-kube-api-access-79gl9\") pod \"cluster-monitoring-operator-884576c57-p7hdq\" (UID: \"fef674c6-31e4-4f47-b723-19bc2b38502c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.491848 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.491813 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" Nov 03 03:01:06.492055 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.492023 5117 scope.go:122] "RemoveContainer" containerID="e08d108d1e4c3e865addb1011469ffd6b7439dd514415798897fcd17f9adbaf6" Nov 03 03:01:06.493137 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.493118 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" Nov 03 03:01:06.493534 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.493515 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:06.505066 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.505026 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:06.505917 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.505537 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:06.872052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.872008 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:06.872052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.872052 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.872283 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.872120 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:06.872283 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.872165 5117 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 03:01:06.872283 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.872230 5117 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:01:06.872283 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.872236 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert podName:5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe nodeName:}" failed. No retries permitted until 2026-11-03 03:01:07.872221346 +0000 UTC m=+2.902819882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-bhczt" (UID: "5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe") : secret "networking-console-plugin-cert" not found Nov 03 03:01:06.872283 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.872275 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:07.872263894 +0000 UTC m=+2.902862410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:01:06.872283 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.872286 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:07.87227984 +0000 UTC m=+2.902878356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:01:06.903892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.903857 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr"] Nov 03 03:01:06.920241 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.920170 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:06.935142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.935115 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:06.935512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.935438 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq"] Nov 03 03:01:06.938599 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:06.938538 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef674c6_31e4_4f47_b723_19bc2b38502c.slice/crio-d90849bd73728961237818a69c94f736271519112e49966fecdcb86634dd5143 WatchSource:0}: Error finding container d90849bd73728961237818a69c94f736271519112e49966fecdcb86634dd5143: Status 404 returned error can't find the container with id d90849bd73728961237818a69c94f736271519112e49966fecdcb86634dd5143 Nov 03 03:01:06.940824 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.940807 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:01:06.952308 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.952275 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675ff945d6-2qfxd"] Nov 03 03:01:06.972553 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.972531 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:06.972631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.972574 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:06.972701 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.972680 5117 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:01:06.972764 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.972753 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:07.972737131 +0000 UTC m=+3.003335647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:01:06.972826 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.972681 5117 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 03:01:06.972870 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:06.972840 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:07.972822433 +0000 UTC m=+3.003420969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : secret "metrics-daemon-secret" not found Nov 03 03:01:06.973170 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:06.973146 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db36b8c_a3dd_4bb0_8710_1d0c09e981cd.slice/crio-37be8d22cf1288e9f5dcd779beab7abc1c376e7fe22a83b6dc17d71e05a1577a WatchSource:0}: Error finding container 37be8d22cf1288e9f5dcd779beab7abc1c376e7fe22a83b6dc17d71e05a1577a: Status 404 returned error can't find the container with id 37be8d22cf1288e9f5dcd779beab7abc1c376e7fe22a83b6dc17d71e05a1577a Nov 03 03:01:06.984641 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.984616 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dwkvw"] Nov 03 03:01:06.988703 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:06.988677 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b7b1fa_14b8_47ee_a3ce_dd236209b96b.slice/crio-f767e0c70f16b0ea64dc46bada6b34860c8a4516abf51094bf4be47a5f42e9bf WatchSource:0}: Error finding container f767e0c70f16b0ea64dc46bada6b34860c8a4516abf51094bf4be47a5f42e9bf: Status 404 returned error can't find the container with id f767e0c70f16b0ea64dc46bada6b34860c8a4516abf51094bf4be47a5f42e9bf Nov 03 03:01:06.993230 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:06.992753 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bf7f94cf9-nvxkf"] Nov 03 03:01:07.004685 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:07.004660 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72608a5a_a7a0_43e7_a9c0_bc681d988262.slice/crio-9b255fd13974d9b3af4d446e95abdfc6b165219638467d3c20a2b24e5b1a7ae1 WatchSource:0}: Error finding container 9b255fd13974d9b3af4d446e95abdfc6b165219638467d3c20a2b24e5b1a7ae1: Status 404 returned error can't find the container with id 9b255fd13974d9b3af4d446e95abdfc6b165219638467d3c20a2b24e5b1a7ae1 Nov 03 03:01:07.156231 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.156125 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dwkvw" event={"ID":"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b","Type":"ContainerStarted","Data":"f767e0c70f16b0ea64dc46bada6b34860c8a4516abf51094bf4be47a5f42e9bf"} Nov 03 03:01:07.157978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.157943 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" event={"ID":"72608a5a-a7a0-43e7-a9c0-bc681d988262","Type":"ContainerStarted","Data":"bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf"} Nov 03 03:01:07.158107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.158002 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" event={"ID":"72608a5a-a7a0-43e7-a9c0-bc681d988262","Type":"ContainerStarted","Data":"9b255fd13974d9b3af4d446e95abdfc6b165219638467d3c20a2b24e5b1a7ae1"} Nov 03 03:01:07.159494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.159463 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" event={"ID":"fef674c6-31e4-4f47-b723-19bc2b38502c","Type":"ContainerStarted","Data":"d90849bd73728961237818a69c94f736271519112e49966fecdcb86634dd5143"} Nov 03 03:01:07.160766 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.160739 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" event={"ID":"2469c20c-30a7-41bb-b726-d62d9324f693","Type":"ContainerStarted","Data":"8a005321af122e71263ca4ca876dd08bd470a60a81edea53166f8bf169df594b"} Nov 03 03:01:07.162546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.162514 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" event={"ID":"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd","Type":"ContainerStarted","Data":"856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f"} Nov 03 03:01:07.162655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.162568 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" event={"ID":"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd","Type":"ContainerStarted","Data":"37be8d22cf1288e9f5dcd779beab7abc1c376e7fe22a83b6dc17d71e05a1577a"} Nov 03 03:01:07.164306 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.164283 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:01:07.164765 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.164751 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/2.log" Nov 03 03:01:07.164843 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.164784 5117 generic.go:356] "Generic (PLEG): container finished" podID="297e0916-b3bb-4002-810c-390a724cd782" containerID="8054f88792884d67702dac77fbb2434fd861e44756ea55b21407f8325acccb96" exitCode=255 Nov 03 03:01:07.164921 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.164898 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" event={"ID":"297e0916-b3bb-4002-810c-390a724cd782","Type":"ContainerDied","Data":"8054f88792884d67702dac77fbb2434fd861e44756ea55b21407f8325acccb96"} Nov 03 03:01:07.164981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.164960 5117 scope.go:122] "RemoveContainer" containerID="e08d108d1e4c3e865addb1011469ffd6b7439dd514415798897fcd17f9adbaf6" Nov 03 03:01:07.165203 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.165133 5117 scope.go:122] "RemoveContainer" containerID="8054f88792884d67702dac77fbb2434fd861e44756ea55b21407f8325acccb96" Nov 03 03:01:07.165809 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.165369 5117 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-f869c_openshift-console-operator(297e0916-b3bb-4002-810c-390a724cd782)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" Nov 03 03:01:07.884021 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.883975 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:07.884218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.884067 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:07.884218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.884087 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:07.884218 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.884163 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:09.884139139 +0000 UTC m=+4.914737676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : configmap references non-existent config key: service-ca.crt Nov 03 03:01:07.884218 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.884198 5117 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 03:01:07.884423 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.884245 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs podName:c02e7385-46cc-46a6-822f-f47fe2819a3f nodeName:}" failed. No retries permitted until 2026-11-03 03:01:09.88423206 +0000 UTC m=+4.914830599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs") pod "router-default-c5c7c7bf7-x5tt5" (UID: "c02e7385-46cc-46a6-822f-f47fe2819a3f") : secret "router-metrics-certs-default" not found Nov 03 03:01:07.886792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.886770 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-bhczt\" (UID: \"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:07.985012 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.984970 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:07.985176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.985024 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:07.985176 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.985141 5117 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 03:01:07.985176 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.985155 5117 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 03:01:07.985319 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.985223 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs podName:1714a90e-d9bd-4a76-bc61-295f6978bf17 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:09.985202155 +0000 UTC m=+5.015800688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs") pod "network-metrics-daemon-xw724" (UID: "1714a90e-d9bd-4a76-bc61-295f6978bf17") : secret "metrics-daemon-secret" not found Nov 03 03:01:07.985319 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:07.985244 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls podName:9644b939-f36f-49ec-8388-c0e2c5bdce5a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:09.98523404 +0000 UTC m=+5.015832565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-762ml" (UID: "9644b939-f36f-49ec-8388-c0e2c5bdce5a") : secret "insights-runtime-extractor-tls" not found Nov 03 03:01:07.995335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:07.995311 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" Nov 03 03:01:08.169930 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:08.169857 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:01:08.170310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:08.169983 5117 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 03:01:08.170310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:08.170195 5117 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 03:01:08.170310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:08.170260 5117 scope.go:122] "RemoveContainer" containerID="8054f88792884d67702dac77fbb2434fd861e44756ea55b21407f8325acccb96" Nov 03 03:01:08.170511 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:08.170492 5117 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-f869c_openshift-console-operator(297e0916-b3bb-4002-810c-390a724cd782)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" Nov 03 03:01:08.365133 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:08.365098 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-bhczt"] Nov 03 03:01:08.398940 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:08.398902 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecc18d6_95be_47f1_b7d3_c2e2b83a28fe.slice/crio-d54ee7e4a830843cda1f316d3ac40e511ca88f33afae1f0089a6a3a91f22e8bc WatchSource:0}: Error finding container d54ee7e4a830843cda1f316d3ac40e511ca88f33afae1f0089a6a3a91f22e8bc: Status 404 returned error can't find the container with id d54ee7e4a830843cda1f316d3ac40e511ca88f33afae1f0089a6a3a91f22e8bc Nov 03 03:01:09.177128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:09.177055 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" event={"ID":"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe","Type":"ContainerStarted","Data":"d54ee7e4a830843cda1f316d3ac40e511ca88f33afae1f0089a6a3a91f22e8bc"} Nov 03 03:01:09.905475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:09.905399 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:09.905741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:09.905531 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:09.906607 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:09.906574 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02e7385-46cc-46a6-822f-f47fe2819a3f-service-ca-bundle\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:09.908612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:09.908587 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c02e7385-46cc-46a6-822f-f47fe2819a3f-metrics-certs\") pod \"router-default-c5c7c7bf7-x5tt5\" (UID: \"c02e7385-46cc-46a6-822f-f47fe2819a3f\") " pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:10.006419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.006384 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:10.006673 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.006427 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:10.009999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.009970 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1714a90e-d9bd-4a76-bc61-295f6978bf17-metrics-certs\") pod \"network-metrics-daemon-xw724\" (UID: \"1714a90e-d9bd-4a76-bc61-295f6978bf17\") " pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:10.010813 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.010785 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9644b939-f36f-49ec-8388-c0e2c5bdce5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-762ml\" (UID: \"9644b939-f36f-49ec-8388-c0e2c5bdce5a\") " pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:10.092509 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.092466 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:10.101612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.101338 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xw724" Nov 03 03:01:10.107181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.107153 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-762ml" Nov 03 03:01:10.197168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.196433 5117 generic.go:356] "Generic (PLEG): container finished" podID="6b7adb9e-2bb1-47f2-a364-04d5ec990c3a" containerID="a1541af6bd3d2b03eee1f7eb605e562f2d80c8a6ad39dab088f6f389c4af8906" exitCode=0 Nov 03 03:01:10.197168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.196584 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" event={"ID":"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a","Type":"ContainerDied","Data":"a1541af6bd3d2b03eee1f7eb605e562f2d80c8a6ad39dab088f6f389c4af8906"} Nov 03 03:01:10.197168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.197006 5117 scope.go:122] "RemoveContainer" containerID="a1541af6bd3d2b03eee1f7eb605e562f2d80c8a6ad39dab088f6f389c4af8906" Nov 03 03:01:10.599937 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.599902 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xw724"] Nov 03 03:01:10.603922 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:10.603891 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1714a90e_d9bd_4a76_bc61_295f6978bf17.slice/crio-f05c51eb4180b53b01c0a6e09508664280845089a8e0c744d4025d4c17b4162b WatchSource:0}: Error finding container f05c51eb4180b53b01c0a6e09508664280845089a8e0c744d4025d4c17b4162b: Status 404 returned error can't find the container with id f05c51eb4180b53b01c0a6e09508664280845089a8e0c744d4025d4c17b4162b Nov 03 03:01:10.608674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.608634 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-762ml"] Nov 03 03:01:10.609604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.609575 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c5c7c7bf7-x5tt5"] Nov 03 03:01:10.611886 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:10.611860 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9644b939_f36f_49ec_8388_c0e2c5bdce5a.slice/crio-d31cc8305e5deea08d38fa4c54ea2ba3dc3526a43a54d1819db2bc5f4216c9bf WatchSource:0}: Error finding container d31cc8305e5deea08d38fa4c54ea2ba3dc3526a43a54d1819db2bc5f4216c9bf: Status 404 returned error can't find the container with id d31cc8305e5deea08d38fa4c54ea2ba3dc3526a43a54d1819db2bc5f4216c9bf Nov 03 03:01:10.663684 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:10.663631 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02e7385_46cc_46a6_822f_f47fe2819a3f.slice/crio-498b3d9cc67d27083f6d2697dc9504a3b22c520f8785aca4bb1739135921cf56 WatchSource:0}: Error finding container 498b3d9cc67d27083f6d2697dc9504a3b22c520f8785aca4bb1739135921cf56: Status 404 returned error can't find the container with id 498b3d9cc67d27083f6d2697dc9504a3b22c520f8785aca4bb1739135921cf56 Nov 03 03:01:10.741212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.741180 5117 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:10.742055 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:10.741712 5117 scope.go:122] "RemoveContainer" containerID="8054f88792884d67702dac77fbb2434fd861e44756ea55b21407f8325acccb96" Nov 03 03:01:10.742055 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:10.741967 5117 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-f869c_openshift-console-operator(297e0916-b3bb-4002-810c-390a724cd782)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" Nov 03 03:01:11.177815 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.177789 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:11.204800 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.204706 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gph9f" event={"ID":"6b7adb9e-2bb1-47f2-a364-04d5ec990c3a","Type":"ContainerStarted","Data":"55eb0d4b09be271fe0ba2af6dd8bfa34faba710dbf0ac5239e52798eeab63952"} Nov 03 03:01:11.207080 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.207053 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-bhczt" event={"ID":"5ecc18d6-95be-47f1-b7d3-c2e2b83a28fe","Type":"ContainerStarted","Data":"3ea256f138699b7f2bfda21fb3fb06a5c9bee543c3117e3982ceda6c4453fe54"} Nov 03 03:01:11.212917 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.212885 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-762ml" event={"ID":"9644b939-f36f-49ec-8388-c0e2c5bdce5a","Type":"ContainerStarted","Data":"fd7ac6e6651e6625264ded75b73afbb297d2f6f03de9a4d211e1b540e34ba72a"} Nov 03 03:01:11.213070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.212943 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-762ml" event={"ID":"9644b939-f36f-49ec-8388-c0e2c5bdce5a","Type":"ContainerStarted","Data":"d31cc8305e5deea08d38fa4c54ea2ba3dc3526a43a54d1819db2bc5f4216c9bf"} Nov 03 03:01:11.215247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.215219 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xw724" event={"ID":"1714a90e-d9bd-4a76-bc61-295f6978bf17","Type":"ContainerStarted","Data":"f05c51eb4180b53b01c0a6e09508664280845089a8e0c744d4025d4c17b4162b"} Nov 03 03:01:11.217275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.217242 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-p7hdq" event={"ID":"fef674c6-31e4-4f47-b723-19bc2b38502c","Type":"ContainerStarted","Data":"32dd58fe162d70000baa8dfa4c19e0c5049177da145bd273dc8d2fc682df6396"} Nov 03 03:01:11.219810 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.219780 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" event={"ID":"2469c20c-30a7-41bb-b726-d62d9324f693","Type":"ContainerStarted","Data":"9d83a6c1fe1dae061fa3c143b59535f4a19f0261b9d0aa376c6c6a21edef4a22"} Nov 03 03:01:11.219915 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.219818 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" event={"ID":"2469c20c-30a7-41bb-b726-d62d9324f693","Type":"ContainerStarted","Data":"e2b3c6138b1a104cfce552df554a805910a738cd55f99157b9bda0cd058b432b"} Nov 03 03:01:11.221476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.221438 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" event={"ID":"c02e7385-46cc-46a6-822f-f47fe2819a3f","Type":"ContainerStarted","Data":"ecd8888696d50b35df0ded8c916763eb3d8fe93f576f12aa874991a2bd05939a"} Nov 03 03:01:11.221628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:11.221603 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" event={"ID":"c02e7385-46cc-46a6-822f-f47fe2819a3f","Type":"ContainerStarted","Data":"498b3d9cc67d27083f6d2697dc9504a3b22c520f8785aca4bb1739135921cf56"} Nov 03 03:01:12.093216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:12.093152 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:12.096233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:12.096209 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:12.946212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:12.946171 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:12.947639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:12.947618 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c5c7c7bf7-x5tt5" Nov 03 03:01:13.232772 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:13.232683 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-762ml" event={"ID":"9644b939-f36f-49ec-8388-c0e2c5bdce5a","Type":"ContainerStarted","Data":"24722b32e34f492edec2b41431788ced3ce644c47893d978fc1be8b7980c9288"} Nov 03 03:01:13.234533 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:13.234491 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dwkvw" event={"ID":"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b","Type":"ContainerStarted","Data":"ae69bc8d40987f84603f0a067608865c2ec5668c49b4f0877b63322c90717fdc"} Nov 03 03:01:13.234732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:13.234547 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dwkvw" event={"ID":"f2b7b1fa-14b8-47ee-a3ce-dd236209b96b","Type":"ContainerStarted","Data":"6f8d91fbaa47febc12548a2774a5eb95a0ba991f7f67229ca55358ec3a469bd7"} Nov 03 03:01:13.267520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:13.267492 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:13.267744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:13.267729 5117 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 03:01:13.506950 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:13.506869 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qg9cd" Nov 03 03:01:14.240137 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:14.240092 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xw724" event={"ID":"1714a90e-d9bd-4a76-bc61-295f6978bf17","Type":"ContainerStarted","Data":"2a1813e6efdbd0a31213e2f3e23443e569cc724c6e7eddb8c16a5e86bc14e59c"} Nov 03 03:01:14.240570 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:14.240193 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xw724" event={"ID":"1714a90e-d9bd-4a76-bc61-295f6978bf17","Type":"ContainerStarted","Data":"9eae9cbe5b3b47d545ea8ef4ad0774b9247e7bc0b970ed02eb32f8d330f48370"} Nov 03 03:01:14.240570 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:14.240281 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:15.287594 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:15.287563 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:16.255413 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:16.255379 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-762ml" event={"ID":"9644b939-f36f-49ec-8388-c0e2c5bdce5a","Type":"ContainerStarted","Data":"58c932651c7084a5b20698384dd157bb4c792cabb461ee28d1c3bdffd6532683"} Nov 03 03:01:16.320203 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:16.320172 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:01:16.920437 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:16.920408 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:16.920832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:16.920818 5117 scope.go:122] "RemoveContainer" containerID="8054f88792884d67702dac77fbb2434fd861e44756ea55b21407f8325acccb96" Nov 03 03:01:17.261323 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:17.261251 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:01:17.261609 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:17.261371 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" event={"ID":"297e0916-b3bb-4002-810c-390a724cd782","Type":"ContainerStarted","Data":"44880577a6eacd16009e6a51b3f1940ff505a23872e529ff3808871d93f00154"} Nov 03 03:01:17.261936 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:17.261916 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:17.263006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:17.262983 5117 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-f869c container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body= Nov 03 03:01:17.263053 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:17.263031 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" podUID="297e0916-b3bb-4002-810c-390a724cd782" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" Nov 03 03:01:18.271164 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:18.271128 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-f869c" Nov 03 03:01:21.639527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:21.639487 5117 patch_prober.go:29] interesting pod/image-registry-5bf7f94cf9-nvxkf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 03:01:21.639989 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:21.639552 5117 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" podUID="72608a5a-a7a0-43e7-a9c0-bc681d988262" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:01:21.927890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:21.927818 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7glvk" Nov 03 03:01:22.501934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.501897 5117 patch_prober.go:29] interesting pod/image-registry-675ff945d6-2qfxd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 03:01:22.502104 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.501956 5117 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" podUID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:01:22.907982 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.907949 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-2jvzf"] Nov 03 03:01:22.912770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.912749 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:22.915471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.915425 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx"] Nov 03 03:01:22.918839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.918815 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" Nov 03 03:01:22.971302 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.971271 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675ff945d6-2qfxd"] Nov 03 03:01:22.976319 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.976292 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 03:01:22.976777 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.976755 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 03:01:22.977134 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.977106 5117 patch_prober.go:29] interesting pod/image-registry-675ff945d6-2qfxd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 03:01:22.977270 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:22.977155 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" podUID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:01:23.018683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.018658 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xpjsx\"" Nov 03 03:01:23.022486 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.022436 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eda3388-921b-4760-a85a-0726a5bd19a6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lk8lx\" (UID: \"2eda3388-921b-4760-a85a-0726a5bd19a6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" Nov 03 03:01:23.022648 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.022553 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a4cff18-5293-4f19-9fd2-29d5d72de74a-tmp\") pod \"downloads-6dd789c8b7-2jvzf\" (UID: \"4a4cff18-5293-4f19-9fd2-29d5d72de74a\") " pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:23.022881 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.022854 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zdr\" (UniqueName: \"kubernetes.io/projected/4a4cff18-5293-4f19-9fd2-29d5d72de74a-kube-api-access-m2zdr\") pod \"downloads-6dd789c8b7-2jvzf\" (UID: \"4a4cff18-5293-4f19-9fd2-29d5d72de74a\") " pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:23.024844 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.024822 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2jvzf"] Nov 03 03:01:23.047819 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.047792 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 03:01:23.047947 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.047921 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jqp47\"" Nov 03 03:01:23.058744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.058715 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx"] Nov 03 03:01:23.088849 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.088817 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c864c5b76-mpwvd"] Nov 03 03:01:23.092283 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.092259 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.123694 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.123663 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a4cff18-5293-4f19-9fd2-29d5d72de74a-tmp\") pod \"downloads-6dd789c8b7-2jvzf\" (UID: \"4a4cff18-5293-4f19-9fd2-29d5d72de74a\") " pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:23.123694 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.123698 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zdr\" (UniqueName: \"kubernetes.io/projected/4a4cff18-5293-4f19-9fd2-29d5d72de74a-kube-api-access-m2zdr\") pod \"downloads-6dd789c8b7-2jvzf\" (UID: \"4a4cff18-5293-4f19-9fd2-29d5d72de74a\") " pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:23.123888 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.123745 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eda3388-921b-4760-a85a-0726a5bd19a6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lk8lx\" (UID: \"2eda3388-921b-4760-a85a-0726a5bd19a6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" Nov 03 03:01:23.124127 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.124106 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a4cff18-5293-4f19-9fd2-29d5d72de74a-tmp\") pod \"downloads-6dd789c8b7-2jvzf\" (UID: \"4a4cff18-5293-4f19-9fd2-29d5d72de74a\") " pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:23.126348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.126326 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2eda3388-921b-4760-a85a-0726a5bd19a6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lk8lx\" (UID: \"2eda3388-921b-4760-a85a-0726a5bd19a6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" Nov 03 03:01:23.162710 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.162633 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c864c5b76-mpwvd"] Nov 03 03:01:23.193605 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.193572 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zdr\" (UniqueName: \"kubernetes.io/projected/4a4cff18-5293-4f19-9fd2-29d5d72de74a-kube-api-access-m2zdr\") pod \"downloads-6dd789c8b7-2jvzf\" (UID: \"4a4cff18-5293-4f19-9fd2-29d5d72de74a\") " pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:23.224511 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.224478 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:23.224757 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.224736 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2470170-ed4c-4db9-9ae0-7b313e238cf3-ca-trust-extracted\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.224811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.224773 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2470170-ed4c-4db9-9ae0-7b313e238cf3-installation-pull-secrets\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.224811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.224789 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgq9d\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-kube-api-access-sgq9d\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.224935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.224922 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2470170-ed4c-4db9-9ae0-7b313e238cf3-trusted-ca\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.224969 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.224954 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2470170-ed4c-4db9-9ae0-7b313e238cf3-image-registry-private-configuration\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.225004 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.224984 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-bound-sa-token\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.225041 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.225008 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-registry-tls\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.225075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.225052 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2470170-ed4c-4db9-9ae0-7b313e238cf3-registry-certificates\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.231764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.231737 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" Nov 03 03:01:23.326097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326060 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2470170-ed4c-4db9-9ae0-7b313e238cf3-trusted-ca\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.326266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326122 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2470170-ed4c-4db9-9ae0-7b313e238cf3-image-registry-private-configuration\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.326266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326169 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-bound-sa-token\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.326266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326207 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-registry-tls\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.326266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326232 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2470170-ed4c-4db9-9ae0-7b313e238cf3-registry-certificates\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.326537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326287 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2470170-ed4c-4db9-9ae0-7b313e238cf3-ca-trust-extracted\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.326537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326318 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2470170-ed4c-4db9-9ae0-7b313e238cf3-installation-pull-secrets\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.326537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.326344 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgq9d\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-kube-api-access-sgq9d\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.327148 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.327121 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2470170-ed4c-4db9-9ae0-7b313e238cf3-registry-certificates\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.327250 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.327208 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2470170-ed4c-4db9-9ae0-7b313e238cf3-trusted-ca\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.327342 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.327301 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2470170-ed4c-4db9-9ae0-7b313e238cf3-ca-trust-extracted\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.329694 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.329675 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2470170-ed4c-4db9-9ae0-7b313e238cf3-installation-pull-secrets\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.329757 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.329719 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2470170-ed4c-4db9-9ae0-7b313e238cf3-image-registry-private-configuration\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.329792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.329763 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-registry-tls\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.392431 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.392395 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-bound-sa-token\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.392604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.392477 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgq9d\" (UniqueName: \"kubernetes.io/projected/e2470170-ed4c-4db9-9ae0-7b313e238cf3-kube-api-access-sgq9d\") pod \"image-registry-5c864c5b76-mpwvd\" (UID: \"e2470170-ed4c-4db9-9ae0-7b313e238cf3\") " pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.403641 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.403609 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:23.656426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.656346 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx"] Nov 03 03:01:23.659303 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.659276 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2jvzf"] Nov 03 03:01:23.661271 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:23.661193 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eda3388_921b_4760_a85a_0726a5bd19a6.slice/crio-8a86ad8b53b7bcdba2f0d2f3b809440946076aeb96aaaabd02c0cba2a2d0d696 WatchSource:0}: Error finding container 8a86ad8b53b7bcdba2f0d2f3b809440946076aeb96aaaabd02c0cba2a2d0d696: Status 404 returned error can't find the container with id 8a86ad8b53b7bcdba2f0d2f3b809440946076aeb96aaaabd02c0cba2a2d0d696 Nov 03 03:01:23.663580 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:23.663534 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4cff18_5293_4f19_9fd2_29d5d72de74a.slice/crio-3194d6d5bf3c4da9ad6ce5a0f34d5d3a164dd58192ab6656bc0bf5b221c1e961 WatchSource:0}: Error finding container 3194d6d5bf3c4da9ad6ce5a0f34d5d3a164dd58192ab6656bc0bf5b221c1e961: Status 404 returned error can't find the container with id 3194d6d5bf3c4da9ad6ce5a0f34d5d3a164dd58192ab6656bc0bf5b221c1e961 Nov 03 03:01:23.814337 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:23.814171 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c864c5b76-mpwvd"] Nov 03 03:01:23.816978 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:23.816943 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2470170_ed4c_4db9_9ae0_7b313e238cf3.slice/crio-1d699a6ec2b4359a4163501c550fe88610d3f168399c03b21a53df9f7eceb1f6 WatchSource:0}: Error finding container 1d699a6ec2b4359a4163501c550fe88610d3f168399c03b21a53df9f7eceb1f6: Status 404 returned error can't find the container with id 1d699a6ec2b4359a4163501c550fe88610d3f168399c03b21a53df9f7eceb1f6 Nov 03 03:01:24.248042 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:24.248012 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dwkvw" Nov 03 03:01:24.296791 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:24.296753 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" event={"ID":"2eda3388-921b-4760-a85a-0726a5bd19a6","Type":"ContainerStarted","Data":"8a86ad8b53b7bcdba2f0d2f3b809440946076aeb96aaaabd02c0cba2a2d0d696"} Nov 03 03:01:24.297979 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:24.297950 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2jvzf" event={"ID":"4a4cff18-5293-4f19-9fd2-29d5d72de74a","Type":"ContainerStarted","Data":"3194d6d5bf3c4da9ad6ce5a0f34d5d3a164dd58192ab6656bc0bf5b221c1e961"} Nov 03 03:01:24.299425 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:24.299404 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" event={"ID":"e2470170-ed4c-4db9-9ae0-7b313e238cf3","Type":"ContainerStarted","Data":"0c5930c0d47b1cde4e43742da5938d8b98cce2fbb6344538c351f06869d53a95"} Nov 03 03:01:24.299555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:24.299438 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" event={"ID":"e2470170-ed4c-4db9-9ae0-7b313e238cf3","Type":"ContainerStarted","Data":"1d699a6ec2b4359a4163501c550fe88610d3f168399c03b21a53df9f7eceb1f6"} Nov 03 03:01:24.299614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:24.299562 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:24.501700 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:24.501584 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" podStartSLOduration=1.5015677570000001 podStartE2EDuration="1.501567757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:24.500653629 +0000 UTC m=+19.531252168" watchObservedRunningTime="2026-11-03 03:01:24.501567757 +0000 UTC m=+19.532166294" Nov 03 03:01:25.291746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:25.291656 5117 patch_prober.go:29] interesting pod/image-registry-5bf7f94cf9-nvxkf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 03:01:25.291746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:25.291713 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" podUID="72608a5a-a7a0-43e7-a9c0-bc681d988262" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:01:27.315610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:27.315574 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" event={"ID":"2eda3388-921b-4760-a85a-0726a5bd19a6","Type":"ContainerStarted","Data":"df127911d166c210c63ee69c2f568df4d93600c9d46ed143c37fc45447084692"} Nov 03 03:01:27.316026 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:27.315801 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" Nov 03 03:01:27.320489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:27.320467 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" Nov 03 03:01:27.434038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:27.433989 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lk8lx" podStartSLOduration=2.158148071 podStartE2EDuration="5.433976431s" totalImagesPullingTime="3.27582836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:27.433711301 +0000 UTC m=+22.464309838" watchObservedRunningTime="2026-11-03 03:01:27.433976431 +0000 UTC m=+22.464574969" Nov 03 03:01:28.603834 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.603799 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8"] Nov 03 03:01:28.622693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.622664 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.667706 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.667675 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8"] Nov 03 03:01:28.717498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.717466 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 03:01:28.734996 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.734970 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 03:01:28.735109 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.735080 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 03:01:28.767870 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.767840 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7nmlj\"" Nov 03 03:01:28.777198 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.777168 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48843ed4-9fc6-4073-8e2c-419df4be9f66-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.777327 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.777235 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grrg\" (UniqueName: \"kubernetes.io/projected/48843ed4-9fc6-4073-8e2c-419df4be9f66-kube-api-access-5grrg\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.777370 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.777320 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48843ed4-9fc6-4073-8e2c-419df4be9f66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.777409 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.777386 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48843ed4-9fc6-4073-8e2c-419df4be9f66-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.878600 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.878514 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grrg\" (UniqueName: \"kubernetes.io/projected/48843ed4-9fc6-4073-8e2c-419df4be9f66-kube-api-access-5grrg\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.878600 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.878557 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48843ed4-9fc6-4073-8e2c-419df4be9f66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.878600 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.878590 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48843ed4-9fc6-4073-8e2c-419df4be9f66-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.878878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.878650 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48843ed4-9fc6-4073-8e2c-419df4be9f66-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.879722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.879691 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48843ed4-9fc6-4073-8e2c-419df4be9f66-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.881350 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.881325 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48843ed4-9fc6-4073-8e2c-419df4be9f66-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.881480 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.881429 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48843ed4-9fc6-4073-8e2c-419df4be9f66-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.945834 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.945800 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grrg\" (UniqueName: \"kubernetes.io/projected/48843ed4-9fc6-4073-8e2c-419df4be9f66-kube-api-access-5grrg\") pod \"prometheus-operator-9cd89b74c-2pcj8\" (UID: \"48843ed4-9fc6-4073-8e2c-419df4be9f66\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:28.974687 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:28.974645 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" Nov 03 03:01:29.382376 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:29.382315 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8"] Nov 03 03:01:29.385204 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:29.385171 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48843ed4_9fc6_4073_8e2c_419df4be9f66.slice/crio-dbffd6e0201c125dd29a2e64f2e80080e8f7d36e3dfee94ee141fd57069a5e8d WatchSource:0}: Error finding container dbffd6e0201c125dd29a2e64f2e80080e8f7d36e3dfee94ee141fd57069a5e8d: Status 404 returned error can't find the container with id dbffd6e0201c125dd29a2e64f2e80080e8f7d36e3dfee94ee141fd57069a5e8d Nov 03 03:01:30.017994 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:30.017953 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:01:30.330421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:30.330377 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" event={"ID":"48843ed4-9fc6-4073-8e2c-419df4be9f66","Type":"ContainerStarted","Data":"dbffd6e0201c125dd29a2e64f2e80080e8f7d36e3dfee94ee141fd57069a5e8d"} Nov 03 03:01:32.977167 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:32.977138 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:34.351020 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:34.350983 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" event={"ID":"48843ed4-9fc6-4073-8e2c-419df4be9f66","Type":"ContainerStarted","Data":"5d1794fcdf8a14dc8035c224f78348ba596f175d3f4b7638dd453964359d2c27"} Nov 03 03:01:34.351020 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:34.351041 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" event={"ID":"48843ed4-9fc6-4073-8e2c-419df4be9f66","Type":"ContainerStarted","Data":"709a65abf6a77b8da529efbd59038d46c6739fc0b9372aef7240cef3976b95f6"} Nov 03 03:01:35.293065 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:35.293036 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:01:35.527671 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:35.527619 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2pcj8" podStartSLOduration=3.592462265 podStartE2EDuration="7.527602885s" totalImagesPullingTime="3.93514062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:34.4835986 +0000 UTC m=+29.514197137" watchObservedRunningTime="2026-11-03 03:01:35.527602885 +0000 UTC m=+30.558201423" Nov 03 03:01:36.325743 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:36.325141 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ldf8b" Nov 03 03:01:37.634312 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.634225 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vbt5z"] Nov 03 03:01:37.670821 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.670790 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.760645 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.760607 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 03:01:37.786923 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.786856 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ccc7x\"" Nov 03 03:01:37.786923 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.786884 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 03:01:37.787158 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.786944 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 03:01:37.862873 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.862839 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-textfile\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863058 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.862912 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-root\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863058 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.863013 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-wtmp\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863058 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.863042 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-tls\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863162 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.863064 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.863185 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-metrics-client-ca\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.863211 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzdm\" (UniqueName: \"kubernetes.io/projected/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-kube-api-access-rqzdm\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.863232 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-sys\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.863264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.863253 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964235 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964152 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-root\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964235 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964201 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-wtmp\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964275 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-root\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964314 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-tls\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964327 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-wtmp\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964351 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964385 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-metrics-client-ca\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964404 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzdm\" (UniqueName: \"kubernetes.io/projected/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-kube-api-access-rqzdm\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964425 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-sys\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964490 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-sys\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964516 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964553 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-textfile\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.964977 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.964955 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-textfile\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.965105 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.965083 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-metrics-client-ca\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.965178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.965157 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.967460 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.967422 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:37.970886 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:37.970858 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-node-exporter-tls\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:38.035527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:38.035494 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzdm\" (UniqueName: \"kubernetes.io/projected/20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5-kube-api-access-rqzdm\") pod \"node-exporter-vbt5z\" (UID: \"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5\") " pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:38.335190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:38.335156 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbt5z" Nov 03 03:01:38.381549 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:01:38.381506 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e6a330_21fb_48f2_a98c_e0fcbeb8f2a5.slice/crio-078e48ed3c54ef15e74875689a3d690ef039e9777e5b416e52a8d8bffd2be053 WatchSource:0}: Error finding container 078e48ed3c54ef15e74875689a3d690ef039e9777e5b416e52a8d8bffd2be053: Status 404 returned error can't find the container with id 078e48ed3c54ef15e74875689a3d690ef039e9777e5b416e52a8d8bffd2be053 Nov 03 03:01:39.378608 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:39.378564 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbt5z" event={"ID":"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5","Type":"ContainerStarted","Data":"078e48ed3c54ef15e74875689a3d690ef039e9777e5b416e52a8d8bffd2be053"} Nov 03 03:01:41.388844 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:41.388802 5117 generic.go:356] "Generic (PLEG): container finished" podID="20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5" containerID="3cc59f85fbdb255b25749a90007433985f47af4cac1c9521536db6f04ce4008c" exitCode=0 Nov 03 03:01:41.389343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:41.388914 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbt5z" event={"ID":"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5","Type":"ContainerDied","Data":"3cc59f85fbdb255b25749a90007433985f47af4cac1c9521536db6f04ce4008c"} Nov 03 03:01:42.396256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:42.396216 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbt5z" event={"ID":"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5","Type":"ContainerStarted","Data":"14c64ffc095a5109b0fa8c9b77f67d372ba85a0f82acb52d641d82fed463e21d"} Nov 03 03:01:42.396731 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:42.396284 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbt5z" event={"ID":"20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5","Type":"ContainerStarted","Data":"e075590f897be59629dddd7be4c4c60527b325cff0028f722fe52ed529d03740"} Nov 03 03:01:45.308433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:45.308400 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c864c5b76-mpwvd" Nov 03 03:01:45.543430 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:45.543376 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vbt5z" podStartSLOduration=6.057898365 podStartE2EDuration="8.543361606s" totalImagesPullingTime="2.404280717s" totalInitContainerRuntime="81.182524ms" isStatefulPod=true podCreationTimestamp="2026-11-03 03:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:42.524129107 +0000 UTC m=+37.554727646" watchObservedRunningTime="2026-11-03 03:01:45.543361606 +0000 UTC m=+40.573960147" Nov 03 03:01:45.548868 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:45.548833 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bf7f94cf9-nvxkf"] Nov 03 03:01:47.995517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:47.995469 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" podUID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" containerName="registry" containerID="cri-o://856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f" gracePeriod=30 Nov 03 03:01:48.266277 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.266248 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:48.364726 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.364685 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.364914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.364791 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.364914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.364836 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.364914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.364867 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.365083 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.364919 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.365083 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.364976 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.365083 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.365007 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.365083 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.365052 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\") pod \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\" (UID: \"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd\") " Nov 03 03:01:48.366064 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.366028 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:01:48.366204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.366078 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:01:48.368916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.368886 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:01:48.368916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.368893 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "kube-api-access-4b8lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:48.369079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.368965 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:48.369079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.369013 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:01:48.371137 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.371104 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:01:48.375315 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.375257 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted" pod "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" (UID: "9db36b8c-a3dd-4bb0-8710-1d0c09e981cd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:01:48.433044 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.433002 5117 generic.go:356] "Generic (PLEG): container finished" podID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" containerID="856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f" exitCode=0 Nov 03 03:01:48.433472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.433047 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" Nov 03 03:01:48.433472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.433091 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" event={"ID":"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd","Type":"ContainerDied","Data":"856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f"} Nov 03 03:01:48.433472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.433151 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675ff945d6-2qfxd" event={"ID":"9db36b8c-a3dd-4bb0-8710-1d0c09e981cd","Type":"ContainerDied","Data":"37be8d22cf1288e9f5dcd779beab7abc1c376e7fe22a83b6dc17d71e05a1577a"} Nov 03 03:01:48.433472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.433183 5117 scope.go:122] "RemoveContainer" containerID="856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f" Nov 03 03:01:48.451795 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.451627 5117 scope.go:122] "RemoveContainer" containerID="856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f" Nov 03 03:01:48.452953 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:01:48.452924 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f\": container with ID starting with 856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f not found: ID does not exist" containerID="856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f" Nov 03 03:01:48.453082 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.452967 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f"} err="failed to get container status \"856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f\": rpc error: code = NotFound desc = could not find container \"856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f\": container with ID starting with 856ef4d5c8c28e0f33bfe32f710ef7f43ffe9cbed31bf4a0ac668da603d4de2f not found: ID does not exist" Nov 03 03:01:48.466304 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466277 5117 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-bound-sa-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.466304 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466304 5117 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-ca-trust-extracted\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.466540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466322 5117 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-image-registry-private-configuration\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.466540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466334 5117 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-trusted-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.466540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466347 5117 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.466540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466356 5117 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-installation-pull-secrets\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.466540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466367 5117 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-registry-certificates\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.466540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.466376 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8lc\" (UniqueName: \"kubernetes.io/projected/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd-kube-api-access-4b8lc\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:01:48.618838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.618806 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675ff945d6-2qfxd"] Nov 03 03:01:48.636098 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:48.636058 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-675ff945d6-2qfxd"] Nov 03 03:01:49.089058 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:49.089016 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" path="/var/lib/kubelet/pods/9db36b8c-a3dd-4bb0-8710-1d0c09e981cd/volumes" Nov 03 03:01:50.449009 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:50.448922 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2jvzf" event={"ID":"4a4cff18-5293-4f19-9fd2-29d5d72de74a","Type":"ContainerStarted","Data":"b9366226deff5c08efa06e670e179cbb009cc7417a9947236331c3adce99ddc1"} Nov 03 03:01:50.449614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:50.449192 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:50.461987 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:50.461950 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-2jvzf" Nov 03 03:01:50.708306 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:01:50.708200 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-2jvzf" podStartSLOduration=2.712667002 podStartE2EDuration="28.708181047s" totalImagesPullingTime="25.995514045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:50.707073556 +0000 UTC m=+45.737672096" watchObservedRunningTime="2026-11-03 03:01:50.708181047 +0000 UTC m=+45.738779587" Nov 03 03:02:00.013931 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:02:00.013890 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:02:10.571482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.571388 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" podUID="72608a5a-a7a0-43e7-a9c0-bc681d988262" containerName="registry" containerID="cri-o://bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf" gracePeriod=30 Nov 03 03:02:10.824001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.823942 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:02:10.975800 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.975767 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.975985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.975824 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.975985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.975860 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.975985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.975878 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.976140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.976075 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.976209 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.976151 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.976262 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.976209 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.976313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.976273 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\") pod \"72608a5a-a7a0-43e7-a9c0-bc681d988262\" (UID: \"72608a5a-a7a0-43e7-a9c0-bc681d988262\") " Nov 03 03:02:10.976529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.976505 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:02:10.976879 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.976857 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:02:10.978770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.978736 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:02:10.978893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.978869 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:02:10.978992 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.978973 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:02:10.979053 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.979011 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "kube-api-access-ggrcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:02:10.979053 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.979020 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:02:10.983266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:10.983241 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted" pod "72608a5a-a7a0-43e7-a9c0-bc681d988262" (UID: "72608a5a-a7a0-43e7-a9c0-bc681d988262"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:02:11.077274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077203 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrcw\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-kube-api-access-ggrcw\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.077274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077229 5117 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-bound-sa-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.077274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077240 5117 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-certificates\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.077274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077249 5117 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-installation-pull-secrets\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.077274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077260 5117 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/72608a5a-a7a0-43e7-a9c0-bc681d988262-ca-trust-extracted\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.077274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077272 5117 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72608a5a-a7a0-43e7-a9c0-bc681d988262-trusted-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.077580 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077285 5117 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/72608a5a-a7a0-43e7-a9c0-bc681d988262-image-registry-private-configuration\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.077580 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.077299 5117 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/72608a5a-a7a0-43e7-a9c0-bc681d988262-registry-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:11.552656 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.552556 5117 generic.go:356] "Generic (PLEG): container finished" podID="72608a5a-a7a0-43e7-a9c0-bc681d988262" containerID="bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf" exitCode=0 Nov 03 03:02:11.552656 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.552639 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" Nov 03 03:02:11.552840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.552649 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" event={"ID":"72608a5a-a7a0-43e7-a9c0-bc681d988262","Type":"ContainerDied","Data":"bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf"} Nov 03 03:02:11.552840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.552733 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bf7f94cf9-nvxkf" event={"ID":"72608a5a-a7a0-43e7-a9c0-bc681d988262","Type":"ContainerDied","Data":"9b255fd13974d9b3af4d446e95abdfc6b165219638467d3c20a2b24e5b1a7ae1"} Nov 03 03:02:11.552840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.552753 5117 scope.go:122] "RemoveContainer" containerID="bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf" Nov 03 03:02:11.560931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.560911 5117 scope.go:122] "RemoveContainer" containerID="bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf" Nov 03 03:02:11.561205 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:02:11.561182 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf\": container with ID starting with bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf not found: ID does not exist" containerID="bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf" Nov 03 03:02:11.561275 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.561218 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf"} err="failed to get container status \"bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf\": rpc error: code = NotFound desc = could not find container \"bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf\": container with ID starting with bd940a12020cccaff184a3323faf21e21e2f6d354f6550e26310e6bb7fb83acf not found: ID does not exist" Nov 03 03:02:11.721150 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.721110 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bf7f94cf9-nvxkf"] Nov 03 03:02:11.771898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:11.771856 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bf7f94cf9-nvxkf"] Nov 03 03:02:13.088249 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:13.088217 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72608a5a-a7a0-43e7-a9c0-bc681d988262" path="/var/lib/kubelet/pods/72608a5a-a7a0-43e7-a9c0-bc681d988262/volumes" Nov 03 03:02:29.629507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:29.629468 5117 generic.go:356] "Generic (PLEG): container finished" podID="59642d89-3475-44f2-8ddd-c95da8e30f44" containerID="ecaadb79349702f5229ecb8ebe32ec04e10f77772d841dc7ab90ede64f4dcc5f" exitCode=0 Nov 03 03:02:29.629932 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:29.629475 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" event={"ID":"59642d89-3475-44f2-8ddd-c95da8e30f44","Type":"ContainerDied","Data":"ecaadb79349702f5229ecb8ebe32ec04e10f77772d841dc7ab90ede64f4dcc5f"} Nov 03 03:02:29.629932 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:29.629843 5117 scope.go:122] "RemoveContainer" containerID="ecaadb79349702f5229ecb8ebe32ec04e10f77772d841dc7ab90ede64f4dcc5f" Nov 03 03:02:30.010892 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:02:30.010809 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:02:30.635456 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:30.635406 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gxsq4" event={"ID":"59642d89-3475-44f2-8ddd-c95da8e30f44","Type":"ContainerStarted","Data":"48be8a0399813b5cc12866c971eadf6fe10cd2aa98a472dc7883ae460eb35492"} Nov 03 03:02:37.666245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:37.666210 5117 generic.go:356] "Generic (PLEG): container finished" podID="8fa09e76-0e81-4449-a649-1b5705b8d5bd" containerID="16d7111d2c498927c9f37e78a4b2bd9ea841944edb3f6ed7a2c06deb795ed168" exitCode=0 Nov 03 03:02:37.666724 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:37.666297 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2hdcl" event={"ID":"8fa09e76-0e81-4449-a649-1b5705b8d5bd","Type":"ContainerDied","Data":"16d7111d2c498927c9f37e78a4b2bd9ea841944edb3f6ed7a2c06deb795ed168"} Nov 03 03:02:37.666724 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:37.666686 5117 scope.go:122] "RemoveContainer" containerID="16d7111d2c498927c9f37e78a4b2bd9ea841944edb3f6ed7a2c06deb795ed168" Nov 03 03:02:38.673273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:38.673239 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2hdcl" event={"ID":"8fa09e76-0e81-4449-a649-1b5705b8d5bd","Type":"ContainerStarted","Data":"7ce704f61cfe13aaaf8da9b4f5a88041d7d8d5cc70fbbd5ae528b879643ee48d"} Nov 03 03:02:38.680513 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:38.680490 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pr56b-bzv26"] Nov 03 03:02:38.680670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:38.680652 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-pr56b-bzv26" podUID="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" containerName="insights-gathering" containerID="cri-o://7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99" gracePeriod=30 Nov 03 03:02:38.947344 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:38.947320 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:02:39.016912 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.016870 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\") pod \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " Nov 03 03:02:39.017115 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.016958 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\") pod \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " Nov 03 03:02:39.017192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.017119 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\" (UniqueName: \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\") pod \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " Nov 03 03:02:39.017192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.017166 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\") pod \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\" (UID: \"9c4667fe-39d4-430d-8fd4-d96b5288d9d8\") " Nov 03 03:02:39.017313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.017284 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle" pod "9c4667fe-39d4-430d-8fd4-d96b5288d9d8" (UID: "9c4667fe-39d4-430d-8fd4-d96b5288d9d8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:02:39.017363 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.017300 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle" pod "9c4667fe-39d4-430d-8fd4-d96b5288d9d8" (UID: "9c4667fe-39d4-430d-8fd4-d96b5288d9d8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:02:39.017567 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.017546 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path" pod "9c4667fe-39d4-430d-8fd4-d96b5288d9d8" (UID: "9c4667fe-39d4-430d-8fd4-d96b5288d9d8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:02:39.017622 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.017548 5117 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-service-ca-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:39.017622 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.017588 5117 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-trusted-ca-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:39.019526 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.019506 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7" pod "9c4667fe-39d4-430d-8fd4-d96b5288d9d8" (UID: "9c4667fe-39d4-430d-8fd4-d96b5288d9d8"). InnerVolumeSpecName "kube-api-access-k2hr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:02:39.118339 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.118302 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hr7\" (UniqueName: \"kubernetes.io/projected/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-kube-api-access-k2hr7\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:39.118339 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.118342 5117 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9c4667fe-39d4-430d-8fd4-d96b5288d9d8-archives-path\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:02:39.679181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.679143 5117 generic.go:356] "Generic (PLEG): container finished" podID="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" containerID="7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99" exitCode=2 Nov 03 03:02:39.679633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.679194 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-pr56b-bzv26" Nov 03 03:02:39.679633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.679226 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pr56b-bzv26" event={"ID":"9c4667fe-39d4-430d-8fd4-d96b5288d9d8","Type":"ContainerDied","Data":"7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99"} Nov 03 03:02:39.679633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.679276 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-pr56b-bzv26" event={"ID":"9c4667fe-39d4-430d-8fd4-d96b5288d9d8","Type":"ContainerDied","Data":"b2f28c2d1f3024a07d08ee6abbfd3488cac486439d7f6765f7a37e1d32507d6a"} Nov 03 03:02:39.679633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.679296 5117 scope.go:122] "RemoveContainer" containerID="7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99" Nov 03 03:02:39.687497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.687320 5117 scope.go:122] "RemoveContainer" containerID="7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99" Nov 03 03:02:39.687612 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:02:39.687593 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99\": container with ID starting with 7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99 not found: ID does not exist" containerID="7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99" Nov 03 03:02:39.687653 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.687623 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99"} err="failed to get container status \"7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99\": rpc error: code = NotFound desc = could not find container \"7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99\": container with ID starting with 7a47ff95b2d2e92cb950ea1f0761864dcaddd23d9a163f97ada3525f47b48d99 not found: ID does not exist" Nov 03 03:02:39.847266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.847231 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-pr56b-bzv26"] Nov 03 03:02:39.897989 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:39.897948 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-pr56b-bzv26"] Nov 03 03:02:41.087734 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:02:41.087704 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" path="/var/lib/kubelet/pods/9c4667fe-39d4-430d-8fd4-d96b5288d9d8/volumes" Nov 03 03:03:00.011672 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:03:00.011627 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:03:30.010228 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:03:30.010150 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:04:00.011026 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:04:00.010990 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:04:30.009969 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:04:30.009929 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:05:00.009932 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:05:00.009894 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:05:30.011532 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:05:30.011496 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:06:00.010907 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:06:00.010864 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:06:05.100827 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:06:05.100799 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:06:05.101802 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:06:05.101782 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:06:30.010630 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:06:30.010546 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:07:00.010108 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:07:00.010061 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:07:30.011626 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:07:30.011587 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:07:59.340435 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340397 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr"] Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340740 5117 cpu_manager.go:414] "containerMap: removing container" podUID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" containerName="registry" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340751 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" containerName="registry" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340766 5117 cpu_manager.go:414] "containerMap: removing container" podUID="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" containerName="insights-gathering" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340771 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" containerName="insights-gathering" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340781 5117 cpu_manager.go:414] "containerMap: removing container" podUID="72608a5a-a7a0-43e7-a9c0-bc681d988262" containerName="registry" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340786 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72608a5a-a7a0-43e7-a9c0-bc681d988262" containerName="registry" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340848 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db36b8c-a3dd-4bb0-8710-1d0c09e981cd" containerName="registry" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340854 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4667fe-39d4-430d-8fd4-d96b5288d9d8" containerName="insights-gathering" Nov 03 03:07:59.340878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.340861 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="72608a5a-a7a0-43e7-a9c0-bc681d988262" containerName="registry" Nov 03 03:07:59.344020 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.344003 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.400670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.400639 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp"] Nov 03 03:07:59.404197 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.404169 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.450713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.450623 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lv77s\"" Nov 03 03:07:59.450713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.450634 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 03:07:59.451333 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.451314 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 03:07:59.452143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.452125 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr"] Nov 03 03:07:59.453060 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.453042 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq"] Nov 03 03:07:59.456513 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.456499 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.472503 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.472475 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.472805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.472524 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.472805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.472551 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.472805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.472606 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.472805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.472713 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjtt\" (UniqueName: \"kubernetes.io/projected/05dd5cee-88e9-407d-b2f1-a42558a00dea-kube-api-access-bvjtt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.472805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.472756 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qgjx\" (UniqueName: \"kubernetes.io/projected/883fa528-8820-4928-9fce-75c091e63bea-kube-api-access-5qgjx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.501626 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.501595 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2qbbf\"" Nov 03 03:07:59.502770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.502733 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp"] Nov 03 03:07:59.503754 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.503715 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx"] Nov 03 03:07:59.507806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.507772 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq"] Nov 03 03:07:59.507930 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.507911 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.566038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.566007 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx"] Nov 03 03:07:59.574054 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574027 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qgjx\" (UniqueName: \"kubernetes.io/projected/883fa528-8820-4928-9fce-75c091e63bea-kube-api-access-5qgjx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.574232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574059 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.574232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574089 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.574232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574110 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhg9\" (UniqueName: \"kubernetes.io/projected/9ebbb138-3f26-4211-bb30-a84237a9a169-kube-api-access-kxhg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.574232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574202 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.574498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574237 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.574498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574263 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.574498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574305 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.574498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574359 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.574498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574467 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.574768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574518 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjtt\" (UniqueName: \"kubernetes.io/projected/05dd5cee-88e9-407d-b2f1-a42558a00dea-kube-api-access-bvjtt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.574768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574536 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.574768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574631 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.574768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574648 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.574768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574631 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.574768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.574700 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njkh\" (UniqueName: \"kubernetes.io/projected/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-kube-api-access-9njkh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.596349 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.596304 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vm7qx\"" Nov 03 03:07:59.596349 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.596311 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-622xh\"" Nov 03 03:07:59.648348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.648317 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjtt\" (UniqueName: \"kubernetes.io/projected/05dd5cee-88e9-407d-b2f1-a42558a00dea-kube-api-access-bvjtt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.649106 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.649088 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qgjx\" (UniqueName: \"kubernetes.io/projected/883fa528-8820-4928-9fce-75c091e63bea-kube-api-access-5qgjx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.659942 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.659913 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:07:59.675278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675249 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njkh\" (UniqueName: \"kubernetes.io/projected/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-kube-api-access-9njkh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.675382 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675311 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.675382 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675367 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhg9\" (UniqueName: \"kubernetes.io/projected/9ebbb138-3f26-4211-bb30-a84237a9a169-kube-api-access-kxhg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.675502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675410 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.675502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675466 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.675502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675497 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.675837 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675816 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.675907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675850 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.675907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675875 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.675907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.675854 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.721909 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.721871 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:07:59.750769 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.750724 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njkh\" (UniqueName: \"kubernetes.io/projected/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-kube-api-access-9njkh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:07:59.760816 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.760789 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhg9\" (UniqueName: \"kubernetes.io/projected/9ebbb138-3f26-4211-bb30-a84237a9a169-kube-api-access-kxhg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.773677 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.773464 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:07:59.825407 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:07:59.825371 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:08:00.011886 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:08:00.011790 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:08:00.035536 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.035511 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr"] Nov 03 03:08:00.037949 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:08:00.037919 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dd5cee_88e9_407d_b2f1_a42558a00dea.slice/crio-48eb10cb6b107e7aa1c106a8e736d1f3b570dce0dd45df0587c12a538ab0ab7e WatchSource:0}: Error finding container 48eb10cb6b107e7aa1c106a8e736d1f3b570dce0dd45df0587c12a538ab0ab7e: Status 404 returned error can't find the container with id 48eb10cb6b107e7aa1c106a8e736d1f3b570dce0dd45df0587c12a538ab0ab7e Nov 03 03:08:00.039981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.039961 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:08:00.138969 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.138939 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp"] Nov 03 03:08:00.140253 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:08:00.140221 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883fa528_8820_4928_9fce_75c091e63bea.slice/crio-1828fb40e531cec339075206e076cfc2e9d27a9e5126d5c4bf4fa68a0c25a28b WatchSource:0}: Error finding container 1828fb40e531cec339075206e076cfc2e9d27a9e5126d5c4bf4fa68a0c25a28b: Status 404 returned error can't find the container with id 1828fb40e531cec339075206e076cfc2e9d27a9e5126d5c4bf4fa68a0c25a28b Nov 03 03:08:00.168515 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.168480 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq"] Nov 03 03:08:00.170693 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:08:00.170661 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebbb138_3f26_4211_bb30_a84237a9a169.slice/crio-20103a9caba3580009d04af168bb99b9b31e7f8b8ff9831ab31c1e2a87dff5c3 WatchSource:0}: Error finding container 20103a9caba3580009d04af168bb99b9b31e7f8b8ff9831ab31c1e2a87dff5c3: Status 404 returned error can't find the container with id 20103a9caba3580009d04af168bb99b9b31e7f8b8ff9831ab31c1e2a87dff5c3 Nov 03 03:08:00.206992 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.206966 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx"] Nov 03 03:08:00.208378 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:08:00.208349 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cb738e_15f1_4edb_8a1d_a64b421c4d06.slice/crio-215e3ee4f846c40a06b3c9cc09c155faa7d3a26d4c0c2c2591a5f8e1b2b63c4e WatchSource:0}: Error finding container 215e3ee4f846c40a06b3c9cc09c155faa7d3a26d4c0c2c2591a5f8e1b2b63c4e: Status 404 returned error can't find the container with id 215e3ee4f846c40a06b3c9cc09c155faa7d3a26d4c0c2c2591a5f8e1b2b63c4e Nov 03 03:08:00.987826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.987785 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" event={"ID":"a8cb738e-15f1-4edb-8a1d-a64b421c4d06","Type":"ContainerStarted","Data":"215e3ee4f846c40a06b3c9cc09c155faa7d3a26d4c0c2c2591a5f8e1b2b63c4e"} Nov 03 03:08:00.989213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.989185 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" event={"ID":"883fa528-8820-4928-9fce-75c091e63bea","Type":"ContainerStarted","Data":"1828fb40e531cec339075206e076cfc2e9d27a9e5126d5c4bf4fa68a0c25a28b"} Nov 03 03:08:00.990502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.990469 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" event={"ID":"9ebbb138-3f26-4211-bb30-a84237a9a169","Type":"ContainerStarted","Data":"20103a9caba3580009d04af168bb99b9b31e7f8b8ff9831ab31c1e2a87dff5c3"} Nov 03 03:08:00.991915 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:00.991887 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" event={"ID":"05dd5cee-88e9-407d-b2f1-a42558a00dea","Type":"ContainerStarted","Data":"48eb10cb6b107e7aa1c106a8e736d1f3b570dce0dd45df0587c12a538ab0ab7e"} Nov 03 03:08:10.056374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.056288 5117 generic.go:356] "Generic (PLEG): container finished" podID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerID="44a296ca3015f0762962ee9bf10707d24ae16723eb02f7861d316284f74b3ddc" exitCode=0 Nov 03 03:08:10.056876 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.056378 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" event={"ID":"05dd5cee-88e9-407d-b2f1-a42558a00dea","Type":"ContainerDied","Data":"44a296ca3015f0762962ee9bf10707d24ae16723eb02f7861d316284f74b3ddc"} Nov 03 03:08:10.058372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.058343 5117 generic.go:356] "Generic (PLEG): container finished" podID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerID="624b67b1fd7c9027ad35d57f08eb257d4430b57f890d796025a28effb692cbc6" exitCode=0 Nov 03 03:08:10.058520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.058467 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" event={"ID":"a8cb738e-15f1-4edb-8a1d-a64b421c4d06","Type":"ContainerDied","Data":"624b67b1fd7c9027ad35d57f08eb257d4430b57f890d796025a28effb692cbc6"} Nov 03 03:08:10.060116 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.060094 5117 generic.go:356] "Generic (PLEG): container finished" podID="883fa528-8820-4928-9fce-75c091e63bea" containerID="cc5ebba397a56bc19a07b4f609a856d14717d9361a0e67e3309928fea4200990" exitCode=0 Nov 03 03:08:10.060235 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.060197 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" event={"ID":"883fa528-8820-4928-9fce-75c091e63bea","Type":"ContainerDied","Data":"cc5ebba397a56bc19a07b4f609a856d14717d9361a0e67e3309928fea4200990"} Nov 03 03:08:10.061978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.061896 5117 generic.go:356] "Generic (PLEG): container finished" podID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerID="63cf21e64cfd2a60c5ea5fa0470bfb9ec618a856b18203266467e84a7842c187" exitCode=0 Nov 03 03:08:10.061978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:10.061938 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" event={"ID":"9ebbb138-3f26-4211-bb30-a84237a9a169","Type":"ContainerDied","Data":"63cf21e64cfd2a60c5ea5fa0470bfb9ec618a856b18203266467e84a7842c187"} Nov 03 03:08:12.073133 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.073100 5117 generic.go:356] "Generic (PLEG): container finished" podID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerID="2d3dafb6bb08b4c06b6ceaf7b10dc88058d415bb6ee6302aaa00e4d91073804c" exitCode=0 Nov 03 03:08:12.073597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.073188 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" event={"ID":"a8cb738e-15f1-4edb-8a1d-a64b421c4d06","Type":"ContainerDied","Data":"2d3dafb6bb08b4c06b6ceaf7b10dc88058d415bb6ee6302aaa00e4d91073804c"} Nov 03 03:08:12.075071 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.075052 5117 generic.go:356] "Generic (PLEG): container finished" podID="883fa528-8820-4928-9fce-75c091e63bea" containerID="5c3a95653a390cdb1821871e54b2a2b88426a4dd330bd5c20203eefb5d52abdf" exitCode=0 Nov 03 03:08:12.075165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.075138 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" event={"ID":"883fa528-8820-4928-9fce-75c091e63bea","Type":"ContainerDied","Data":"5c3a95653a390cdb1821871e54b2a2b88426a4dd330bd5c20203eefb5d52abdf"} Nov 03 03:08:12.076916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.076898 5117 generic.go:356] "Generic (PLEG): container finished" podID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerID="78f4d8581f38ee671aeaa0b68a4ac4cd154060be68e8757313c7aab7ed7bb1b7" exitCode=0 Nov 03 03:08:12.077019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.077004 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" event={"ID":"9ebbb138-3f26-4211-bb30-a84237a9a169","Type":"ContainerDied","Data":"78f4d8581f38ee671aeaa0b68a4ac4cd154060be68e8757313c7aab7ed7bb1b7"} Nov 03 03:08:12.078777 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.078759 5117 generic.go:356] "Generic (PLEG): container finished" podID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerID="ecb915a3d60054a7ca2cd0cb28979145441eec6445258d92199aa227ff9b73b4" exitCode=0 Nov 03 03:08:12.078915 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:12.078819 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" event={"ID":"05dd5cee-88e9-407d-b2f1-a42558a00dea","Type":"ContainerDied","Data":"ecb915a3d60054a7ca2cd0cb28979145441eec6445258d92199aa227ff9b73b4"} Nov 03 03:08:23.151828 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:23.151786 5117 generic.go:356] "Generic (PLEG): container finished" podID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerID="772cdc888e02e22380ed7e51f04d4b5b68d8acf2ec9af824743ca20cad48b85a" exitCode=0 Nov 03 03:08:23.152387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:23.151943 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" event={"ID":"05dd5cee-88e9-407d-b2f1-a42558a00dea","Type":"ContainerDied","Data":"772cdc888e02e22380ed7e51f04d4b5b68d8acf2ec9af824743ca20cad48b85a"} Nov 03 03:08:23.155131 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:23.155100 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" event={"ID":"a8cb738e-15f1-4edb-8a1d-a64b421c4d06","Type":"ContainerStarted","Data":"695c9a8f27e24b5887a40f4a0e535a5a68fbdf0aea5f2f699476c0b721f7e54b"} Nov 03 03:08:23.158052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:23.158021 5117 generic.go:356] "Generic (PLEG): container finished" podID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerID="c5260e75f3b2457c644990968e3e2767b3aa4cd4f0471acc80676c401429fbcf" exitCode=0 Nov 03 03:08:23.158241 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:23.158049 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" event={"ID":"9ebbb138-3f26-4211-bb30-a84237a9a169","Type":"ContainerDied","Data":"c5260e75f3b2457c644990968e3e2767b3aa4cd4f0471acc80676c401429fbcf"} Nov 03 03:08:24.164577 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.164488 5117 generic.go:356] "Generic (PLEG): container finished" podID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerID="695c9a8f27e24b5887a40f4a0e535a5a68fbdf0aea5f2f699476c0b721f7e54b" exitCode=0 Nov 03 03:08:24.165035 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.164581 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" event={"ID":"a8cb738e-15f1-4edb-8a1d-a64b421c4d06","Type":"ContainerDied","Data":"695c9a8f27e24b5887a40f4a0e535a5a68fbdf0aea5f2f699476c0b721f7e54b"} Nov 03 03:08:24.166701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.166672 5117 generic.go:356] "Generic (PLEG): container finished" podID="883fa528-8820-4928-9fce-75c091e63bea" containerID="06d474163acefdc06cf53a4c122120e5c83f3dc14f4b14a71969e3bb35e48697" exitCode=0 Nov 03 03:08:24.166817 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.166713 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" event={"ID":"883fa528-8820-4928-9fce-75c091e63bea","Type":"ContainerDied","Data":"06d474163acefdc06cf53a4c122120e5c83f3dc14f4b14a71969e3bb35e48697"} Nov 03 03:08:24.298276 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.298248 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:08:24.316701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.316674 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:08:24.405118 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405077 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebbb138-3f26-4211-bb30-a84237a9a169-kube-api-access-kxhg9\" (UniqueName: \"kubernetes.io/projected/9ebbb138-3f26-4211-bb30-a84237a9a169-kube-api-access-kxhg9\") pod \"9ebbb138-3f26-4211-bb30-a84237a9a169\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " Nov 03 03:08:24.405329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405129 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-util\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-util\") pod \"05dd5cee-88e9-407d-b2f1-a42558a00dea\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " Nov 03 03:08:24.405329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405161 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-bundle\") pod \"9ebbb138-3f26-4211-bb30-a84237a9a169\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " Nov 03 03:08:24.405329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405224 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-util\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-util\") pod \"9ebbb138-3f26-4211-bb30-a84237a9a169\" (UID: \"9ebbb138-3f26-4211-bb30-a84237a9a169\") " Nov 03 03:08:24.405329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405301 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-bundle\") pod \"05dd5cee-88e9-407d-b2f1-a42558a00dea\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " Nov 03 03:08:24.405593 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405345 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dd5cee-88e9-407d-b2f1-a42558a00dea-kube-api-access-bvjtt\" (UniqueName: \"kubernetes.io/projected/05dd5cee-88e9-407d-b2f1-a42558a00dea-kube-api-access-bvjtt\") pod \"05dd5cee-88e9-407d-b2f1-a42558a00dea\" (UID: \"05dd5cee-88e9-407d-b2f1-a42558a00dea\") " Nov 03 03:08:24.405849 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405811 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-bundle" pod "05dd5cee-88e9-407d-b2f1-a42558a00dea" (UID: "05dd5cee-88e9-407d-b2f1-a42558a00dea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:24.406125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.405945 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-bundle" pod "9ebbb138-3f26-4211-bb30-a84237a9a169" (UID: "9ebbb138-3f26-4211-bb30-a84237a9a169"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:24.407754 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.407732 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebbb138-3f26-4211-bb30-a84237a9a169-kube-api-access-kxhg9" pod "9ebbb138-3f26-4211-bb30-a84237a9a169" (UID: "9ebbb138-3f26-4211-bb30-a84237a9a169"). InnerVolumeSpecName "kube-api-access-kxhg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:24.407912 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.407896 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dd5cee-88e9-407d-b2f1-a42558a00dea-kube-api-access-bvjtt" pod "05dd5cee-88e9-407d-b2f1-a42558a00dea" (UID: "05dd5cee-88e9-407d-b2f1-a42558a00dea"). InnerVolumeSpecName "kube-api-access-bvjtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:24.411552 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.411529 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-util" pod "05dd5cee-88e9-407d-b2f1-a42558a00dea" (UID: "05dd5cee-88e9-407d-b2f1-a42558a00dea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:24.413257 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.413233 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-util" pod "9ebbb138-3f26-4211-bb30-a84237a9a169" (UID: "9ebbb138-3f26-4211-bb30-a84237a9a169"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:24.506410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.506364 5117 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:24.506410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.506398 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjtt\" (UniqueName: \"kubernetes.io/projected/05dd5cee-88e9-407d-b2f1-a42558a00dea-kube-api-access-bvjtt\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:24.506410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.506409 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhg9\" (UniqueName: \"kubernetes.io/projected/9ebbb138-3f26-4211-bb30-a84237a9a169-kube-api-access-kxhg9\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:24.506410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.506417 5117 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd5cee-88e9-407d-b2f1-a42558a00dea-util\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:24.506718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.506427 5117 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:24.506718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:24.506435 5117 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ebbb138-3f26-4211-bb30-a84237a9a169-util\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:25.173555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.173525 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" Nov 03 03:08:25.173555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.173542 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn9jsq" event={"ID":"9ebbb138-3f26-4211-bb30-a84237a9a169","Type":"ContainerDied","Data":"20103a9caba3580009d04af168bb99b9b31e7f8b8ff9831ab31c1e2a87dff5c3"} Nov 03 03:08:25.174054 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.173601 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20103a9caba3580009d04af168bb99b9b31e7f8b8ff9831ab31c1e2a87dff5c3" Nov 03 03:08:25.175755 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.175729 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" Nov 03 03:08:25.175899 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.175766 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678gxr" event={"ID":"05dd5cee-88e9-407d-b2f1-a42558a00dea","Type":"ContainerDied","Data":"48eb10cb6b107e7aa1c106a8e736d1f3b570dce0dd45df0587c12a538ab0ab7e"} Nov 03 03:08:25.175899 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.175827 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48eb10cb6b107e7aa1c106a8e736d1f3b570dce0dd45df0587c12a538ab0ab7e" Nov 03 03:08:25.322787 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.322762 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:08:25.350454 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.350369 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:08:25.415211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.415174 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-kube-api-access-9njkh\" (UniqueName: \"kubernetes.io/projected/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-kube-api-access-9njkh\") pod \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " Nov 03 03:08:25.415414 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.415238 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-util\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-util\") pod \"883fa528-8820-4928-9fce-75c091e63bea\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " Nov 03 03:08:25.415414 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.415275 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883fa528-8820-4928-9fce-75c091e63bea-kube-api-access-5qgjx\" (UniqueName: \"kubernetes.io/projected/883fa528-8820-4928-9fce-75c091e63bea-kube-api-access-5qgjx\") pod \"883fa528-8820-4928-9fce-75c091e63bea\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " Nov 03 03:08:25.415588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.415434 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-bundle\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-bundle\") pod \"883fa528-8820-4928-9fce-75c091e63bea\" (UID: \"883fa528-8820-4928-9fce-75c091e63bea\") " Nov 03 03:08:25.415588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.415524 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-bundle\") pod \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " Nov 03 03:08:25.415588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.415571 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-util\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-util\") pod \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\" (UID: \"a8cb738e-15f1-4edb-8a1d-a64b421c4d06\") " Nov 03 03:08:25.416085 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.416057 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-bundle" pod "883fa528-8820-4928-9fce-75c091e63bea" (UID: "883fa528-8820-4928-9fce-75c091e63bea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:25.416597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.416572 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-bundle" pod "a8cb738e-15f1-4edb-8a1d-a64b421c4d06" (UID: "a8cb738e-15f1-4edb-8a1d-a64b421c4d06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:25.417696 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.417674 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883fa528-8820-4928-9fce-75c091e63bea-kube-api-access-5qgjx" pod "883fa528-8820-4928-9fce-75c091e63bea" (UID: "883fa528-8820-4928-9fce-75c091e63bea"). InnerVolumeSpecName "kube-api-access-5qgjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:25.417815 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.417798 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-kube-api-access-9njkh" pod "a8cb738e-15f1-4edb-8a1d-a64b421c4d06" (UID: "a8cb738e-15f1-4edb-8a1d-a64b421c4d06"). InnerVolumeSpecName "kube-api-access-9njkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:08:25.421083 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.421059 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-util" pod "883fa528-8820-4928-9fce-75c091e63bea" (UID: "883fa528-8820-4928-9fce-75c091e63bea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:25.423086 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.423066 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-util" pod "a8cb738e-15f1-4edb-8a1d-a64b421c4d06" (UID: "a8cb738e-15f1-4edb-8a1d-a64b421c4d06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:08:25.516811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.516766 5117 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:25.516811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.516797 5117 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-util\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:25.516811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.516808 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njkh\" (UniqueName: \"kubernetes.io/projected/a8cb738e-15f1-4edb-8a1d-a64b421c4d06-kube-api-access-9njkh\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:25.516811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.516817 5117 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-util\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:25.516811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.516825 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qgjx\" (UniqueName: \"kubernetes.io/projected/883fa528-8820-4928-9fce-75c091e63bea-kube-api-access-5qgjx\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:25.517113 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:25.516833 5117 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/883fa528-8820-4928-9fce-75c091e63bea-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:08:26.182235 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:26.182203 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" Nov 03 03:08:26.182235 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:26.182213 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aghwdx" event={"ID":"a8cb738e-15f1-4edb-8a1d-a64b421c4d06","Type":"ContainerDied","Data":"215e3ee4f846c40a06b3c9cc09c155faa7d3a26d4c0c2c2591a5f8e1b2b63c4e"} Nov 03 03:08:26.182830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:26.182262 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="215e3ee4f846c40a06b3c9cc09c155faa7d3a26d4c0c2c2591a5f8e1b2b63c4e" Nov 03 03:08:26.188434 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:26.188399 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" event={"ID":"883fa528-8820-4928-9fce-75c091e63bea","Type":"ContainerDied","Data":"1828fb40e531cec339075206e076cfc2e9d27a9e5126d5c4bf4fa68a0c25a28b"} Nov 03 03:08:26.188649 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:26.188499 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1828fb40e531cec339075206e076cfc2e9d27a9e5126d5c4bf4fa68a0c25a28b" Nov 03 03:08:26.188649 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:26.188605 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z57rp" Nov 03 03:08:30.011204 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:08:30.011168 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:08:31.957009 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.956974 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9szxj"] Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957341 5117 cpu_manager.go:414] "containerMap: removing container" podUID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerName="pull" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957351 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerName="pull" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957364 5117 cpu_manager.go:414] "containerMap: removing container" podUID="883fa528-8820-4928-9fce-75c091e63bea" containerName="pull" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957370 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883fa528-8820-4928-9fce-75c091e63bea" containerName="pull" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957380 5117 cpu_manager.go:414] "containerMap: removing container" podUID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerName="pull" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957385 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerName="pull" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957395 5117 cpu_manager.go:414] "containerMap: removing container" podUID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerName="util" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957400 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerName="util" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957406 5117 cpu_manager.go:414] "containerMap: removing container" podUID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerName="util" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957410 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerName="util" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957417 5117 cpu_manager.go:414] "containerMap: removing container" podUID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerName="extract" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957421 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerName="extract" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957428 5117 cpu_manager.go:414] "containerMap: removing container" podUID="883fa528-8820-4928-9fce-75c091e63bea" containerName="extract" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957433 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883fa528-8820-4928-9fce-75c091e63bea" containerName="extract" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957457 5117 cpu_manager.go:414] "containerMap: removing container" podUID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerName="extract" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957461 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerName="extract" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957478 5117 cpu_manager.go:414] "containerMap: removing container" podUID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerName="util" Nov 03 03:08:31.957438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957482 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerName="util" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957488 5117 cpu_manager.go:414] "containerMap: removing container" podUID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerName="extract" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957492 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerName="extract" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957503 5117 cpu_manager.go:414] "containerMap: removing container" podUID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerName="pull" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957507 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerName="pull" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957515 5117 cpu_manager.go:414] "containerMap: removing container" podUID="883fa528-8820-4928-9fce-75c091e63bea" containerName="util" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957519 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883fa528-8820-4928-9fce-75c091e63bea" containerName="util" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957577 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cb738e-15f1-4edb-8a1d-a64b421c4d06" containerName="extract" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957583 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="883fa528-8820-4928-9fce-75c091e63bea" containerName="extract" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957590 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dd5cee-88e9-407d-b2f1-a42558a00dea" containerName="extract" Nov 03 03:08:31.958013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.957596 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebbb138-3f26-4211-bb30-a84237a9a169" containerName="extract" Nov 03 03:08:31.959861 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:31.959844 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" Nov 03 03:08:32.049682 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.049645 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bbmff\"" Nov 03 03:08:32.075679 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.075640 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkbl\" (UniqueName: \"kubernetes.io/projected/70c7e0ef-af42-493c-baf8-1322dcbd8074-kube-api-access-llkbl\") pod \"authorino-operator-7b7bb4bc77-9szxj\" (UID: \"70c7e0ef-af42-493c-baf8-1322dcbd8074\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" Nov 03 03:08:32.100686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.100648 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 03:08:32.100686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.100679 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 03:08:32.102206 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.102184 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9szxj"] Nov 03 03:08:32.176863 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.176829 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkbl\" (UniqueName: \"kubernetes.io/projected/70c7e0ef-af42-493c-baf8-1322dcbd8074-kube-api-access-llkbl\") pod \"authorino-operator-7b7bb4bc77-9szxj\" (UID: \"70c7e0ef-af42-493c-baf8-1322dcbd8074\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" Nov 03 03:08:32.256518 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.256427 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkbl\" (UniqueName: \"kubernetes.io/projected/70c7e0ef-af42-493c-baf8-1322dcbd8074-kube-api-access-llkbl\") pod \"authorino-operator-7b7bb4bc77-9szxj\" (UID: \"70c7e0ef-af42-493c-baf8-1322dcbd8074\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" Nov 03 03:08:32.271274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.271250 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" Nov 03 03:08:32.649382 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:32.649357 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-9szxj"] Nov 03 03:08:32.651737 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:08:32.651707 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c7e0ef_af42_493c_baf8_1322dcbd8074.slice/crio-e56304992f9af0c21945e127f895de1089144a85bc80123d32456b529ecdd2e5 WatchSource:0}: Error finding container e56304992f9af0c21945e127f895de1089144a85bc80123d32456b529ecdd2e5: Status 404 returned error can't find the container with id e56304992f9af0c21945e127f895de1089144a85bc80123d32456b529ecdd2e5 Nov 03 03:08:33.219455 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:33.219418 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" event={"ID":"70c7e0ef-af42-493c-baf8-1322dcbd8074","Type":"ContainerStarted","Data":"e56304992f9af0c21945e127f895de1089144a85bc80123d32456b529ecdd2e5"} Nov 03 03:08:38.243651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:38.243604 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" event={"ID":"70c7e0ef-af42-493c-baf8-1322dcbd8074","Type":"ContainerStarted","Data":"64d0af01be0de9444d5a008f55b1ac3d16502b0f3539d55ed0fb3e89874ac06f"} Nov 03 03:08:38.244139 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:38.243710 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" Nov 03 03:08:38.413620 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:38.413560 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" podStartSLOduration=2.559093275 podStartE2EDuration="7.413545544s" totalImagesPullingTime="4.854452269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:38.411289494 +0000 UTC m=+453.441888047" watchObservedRunningTime="2026-11-03 03:08:38.413545544 +0000 UTC m=+453.444144083" Nov 03 03:08:40.086904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.086876 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr"] Nov 03 03:08:40.090235 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.090218 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.146505 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.146463 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6j4\" (UniqueName: \"kubernetes.io/projected/85e920ac-42c2-4f5e-9aba-9b282f92611e-kube-api-access-wf6j4\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.146718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.146620 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.146718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.146661 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.146718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.146690 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85e920ac-42c2-4f5e-9aba-9b282f92611e-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.146867 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.146828 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.146914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.146893 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85e920ac-42c2-4f5e-9aba-9b282f92611e-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.216297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.216265 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr"] Nov 03 03:08:40.221006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.220984 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 03:08:40.221136 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.221028 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 03:08:40.221222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.221206 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 03:08:40.221289 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.221277 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-slhgs\"" Nov 03 03:08:40.227143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.227129 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 03:08:40.227494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.227482 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 03:08:40.247543 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.247517 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.247659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.247546 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.247659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.247567 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85e920ac-42c2-4f5e-9aba-9b282f92611e-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.247659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.247616 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.247659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.247642 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85e920ac-42c2-4f5e-9aba-9b282f92611e-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.247844 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.247684 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6j4\" (UniqueName: \"kubernetes.io/projected/85e920ac-42c2-4f5e-9aba-9b282f92611e-kube-api-access-wf6j4\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.248143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.248114 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85e920ac-42c2-4f5e-9aba-9b282f92611e-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.250296 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.250271 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85e920ac-42c2-4f5e-9aba-9b282f92611e-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.250677 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.250656 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.250734 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.250682 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.250734 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.250690 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85e920ac-42c2-4f5e-9aba-9b282f92611e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.327683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.327649 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6j4\" (UniqueName: \"kubernetes.io/projected/85e920ac-42c2-4f5e-9aba-9b282f92611e-kube-api-access-wf6j4\") pod \"opendatahub-operator-controller-manager-69c4cfd777-nw6wr\" (UID: \"85e920ac-42c2-4f5e-9aba-9b282f92611e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.405069 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.404974 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:40.936336 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:08:40.936307 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e920ac_42c2_4f5e_9aba_9b282f92611e.slice/crio-60d2993fbba359e719b922f843959c729566fe3e7f592b993fa66f607ba4ef74 WatchSource:0}: Error finding container 60d2993fbba359e719b922f843959c729566fe3e7f592b993fa66f607ba4ef74: Status 404 returned error can't find the container with id 60d2993fbba359e719b922f843959c729566fe3e7f592b993fa66f607ba4ef74 Nov 03 03:08:40.949573 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:40.949543 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr"] Nov 03 03:08:41.259329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:41.259253 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" event={"ID":"85e920ac-42c2-4f5e-9aba-9b282f92611e","Type":"ContainerStarted","Data":"60d2993fbba359e719b922f843959c729566fe3e7f592b993fa66f607ba4ef74"} Nov 03 03:08:46.287955 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:46.287866 5117 generic.go:356] "Generic (PLEG): container finished" podID="85e920ac-42c2-4f5e-9aba-9b282f92611e" containerID="66a15803d404ef3806851ef77ac650cd0662db035e65f47bbefe33ab561c1ae6" exitCode=0 Nov 03 03:08:46.288389 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:46.287960 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" event={"ID":"85e920ac-42c2-4f5e-9aba-9b282f92611e","Type":"ContainerDied","Data":"66a15803d404ef3806851ef77ac650cd0662db035e65f47bbefe33ab561c1ae6"} Nov 03 03:08:47.294772 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:47.294734 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" event={"ID":"85e920ac-42c2-4f5e-9aba-9b282f92611e","Type":"ContainerStarted","Data":"93629d6aa1d915ffa5f0b2220c1583b849d652cab0fe448e0e09617c29b458b6"} Nov 03 03:08:47.295195 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:47.294824 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:08:47.579350 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:47.579294 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" podStartSLOduration=2.765449899 podStartE2EDuration="8.579279943s" totalImagesPullingTime="5.32969336s" totalInitContainerRuntime="484.136684ms" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:47.527789689 +0000 UTC m=+462.558388237" watchObservedRunningTime="2026-11-03 03:08:47.579279943 +0000 UTC m=+462.609878520" Nov 03 03:08:49.251497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:49.251464 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-9szxj" Nov 03 03:08:56.045124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.045091 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q"] Nov 03 03:08:56.092854 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.092819 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q"] Nov 03 03:08:56.093011 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.092938 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.153154 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.153116 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 03:08:56.153154 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.153139 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 03:08:56.153370 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.153210 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6jbzr\"" Nov 03 03:08:56.190375 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.190324 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mvx\" (UniqueName: \"kubernetes.io/projected/8e43d17d-d075-43b3-b91e-6b0e494b18d4-kube-api-access-x6mvx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q\" (UID: \"8e43d17d-d075-43b3-b91e-6b0e494b18d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.190573 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.190438 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e43d17d-d075-43b3-b91e-6b0e494b18d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q\" (UID: \"8e43d17d-d075-43b3-b91e-6b0e494b18d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.291213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.291173 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mvx\" (UniqueName: \"kubernetes.io/projected/8e43d17d-d075-43b3-b91e-6b0e494b18d4-kube-api-access-x6mvx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q\" (UID: \"8e43d17d-d075-43b3-b91e-6b0e494b18d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.291415 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.291232 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e43d17d-d075-43b3-b91e-6b0e494b18d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q\" (UID: \"8e43d17d-d075-43b3-b91e-6b0e494b18d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.291630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.291614 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e43d17d-d075-43b3-b91e-6b0e494b18d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q\" (UID: \"8e43d17d-d075-43b3-b91e-6b0e494b18d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.357666 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.357635 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mvx\" (UniqueName: \"kubernetes.io/projected/8e43d17d-d075-43b3-b91e-6b0e494b18d4-kube-api-access-x6mvx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q\" (UID: \"8e43d17d-d075-43b3-b91e-6b0e494b18d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.429094 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.429059 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" Nov 03 03:08:56.932401 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:56.932376 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q"] Nov 03 03:08:56.935409 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:08:56.935383 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e43d17d_d075_43b3_b91e_6b0e494b18d4.slice/crio-f057c29bde7bf0932adb9d6bede086113ba80287bd07da16234b47a844ea61fa WatchSource:0}: Error finding container f057c29bde7bf0932adb9d6bede086113ba80287bd07da16234b47a844ea61fa: Status 404 returned error can't find the container with id f057c29bde7bf0932adb9d6bede086113ba80287bd07da16234b47a844ea61fa Nov 03 03:08:57.353962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:57.353927 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" event={"ID":"8e43d17d-d075-43b3-b91e-6b0e494b18d4","Type":"ContainerStarted","Data":"f057c29bde7bf0932adb9d6bede086113ba80287bd07da16234b47a844ea61fa"} Nov 03 03:08:58.304078 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:08:58.304047 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-nw6wr" Nov 03 03:09:00.020362 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:09:00.020298 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:09:01.375175 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:01.375077 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" event={"ID":"8e43d17d-d075-43b3-b91e-6b0e494b18d4","Type":"ContainerStarted","Data":"51c099a34d15fe88e72ffda2e4edf93347e58614d8a51f55064b6668d036c33d"} Nov 03 03:09:01.609868 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:01.609795 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ldm4q" podStartSLOduration=2.442768899 podStartE2EDuration="6.609779721s" totalImagesPullingTime="4.167010822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:01.609001114 +0000 UTC m=+476.639599655" watchObservedRunningTime="2026-11-03 03:09:01.609779721 +0000 UTC m=+476.640378253" Nov 03 03:09:04.158916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.158883 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9"] Nov 03 03:09:04.162238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.162221 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.220299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.220267 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-x8smk\"" Nov 03 03:09:04.257860 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.257823 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3bb8fd4-ba56-48e2-a299-a5592943018b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kvtg9\" (UID: \"c3bb8fd4-ba56-48e2-a299-a5592943018b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.258039 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.257880 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bg5\" (UniqueName: \"kubernetes.io/projected/c3bb8fd4-ba56-48e2-a299-a5592943018b-kube-api-access-k2bg5\") pod \"cert-manager-webhook-5dbbbf75cc-kvtg9\" (UID: \"c3bb8fd4-ba56-48e2-a299-a5592943018b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.271187 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.271154 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 03:09:04.271315 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.271200 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 03:09:04.272925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.272904 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9"] Nov 03 03:09:04.359161 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.359120 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3bb8fd4-ba56-48e2-a299-a5592943018b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kvtg9\" (UID: \"c3bb8fd4-ba56-48e2-a299-a5592943018b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.359370 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.359173 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bg5\" (UniqueName: \"kubernetes.io/projected/c3bb8fd4-ba56-48e2-a299-a5592943018b-kube-api-access-k2bg5\") pod \"cert-manager-webhook-5dbbbf75cc-kvtg9\" (UID: \"c3bb8fd4-ba56-48e2-a299-a5592943018b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.425590 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.425511 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bg5\" (UniqueName: \"kubernetes.io/projected/c3bb8fd4-ba56-48e2-a299-a5592943018b-kube-api-access-k2bg5\") pod \"cert-manager-webhook-5dbbbf75cc-kvtg9\" (UID: \"c3bb8fd4-ba56-48e2-a299-a5592943018b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.426474 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.426427 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3bb8fd4-ba56-48e2-a299-a5592943018b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-kvtg9\" (UID: \"c3bb8fd4-ba56-48e2-a299-a5592943018b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.485617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.485577 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:04.862165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:04.862139 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9"] Nov 03 03:09:04.864287 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:09:04.864263 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3bb8fd4_ba56_48e2_a299_a5592943018b.slice/crio-00bf57e314a5db89a91f94a9cce3a9a94c16db09b67d730b621f8bf680806de5 WatchSource:0}: Error finding container 00bf57e314a5db89a91f94a9cce3a9a94c16db09b67d730b621f8bf680806de5: Status 404 returned error can't find the container with id 00bf57e314a5db89a91f94a9cce3a9a94c16db09b67d730b621f8bf680806de5 Nov 03 03:09:05.394799 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:05.394762 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" event={"ID":"c3bb8fd4-ba56-48e2-a299-a5592943018b","Type":"ContainerStarted","Data":"00bf57e314a5db89a91f94a9cce3a9a94c16db09b67d730b621f8bf680806de5"} Nov 03 03:09:10.069290 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.069254 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt"] Nov 03 03:09:10.072726 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.072707 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.097832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.097796 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnczq\" (UniqueName: \"kubernetes.io/projected/b1461156-0457-4db2-bc3c-17cc251a2076-kube-api-access-qnczq\") pod \"cert-manager-cainjector-5c57bbd4fc-t4jbt\" (UID: \"b1461156-0457-4db2-bc3c-17cc251a2076\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.098023 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.097862 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1461156-0457-4db2-bc3c-17cc251a2076-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t4jbt\" (UID: \"b1461156-0457-4db2-bc3c-17cc251a2076\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.131532 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.131500 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt"] Nov 03 03:09:10.180261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.180223 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bsd4d\"" Nov 03 03:09:10.198354 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.198319 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnczq\" (UniqueName: \"kubernetes.io/projected/b1461156-0457-4db2-bc3c-17cc251a2076-kube-api-access-qnczq\") pod \"cert-manager-cainjector-5c57bbd4fc-t4jbt\" (UID: \"b1461156-0457-4db2-bc3c-17cc251a2076\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.198543 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.198419 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1461156-0457-4db2-bc3c-17cc251a2076-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t4jbt\" (UID: \"b1461156-0457-4db2-bc3c-17cc251a2076\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.266731 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.266693 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnczq\" (UniqueName: \"kubernetes.io/projected/b1461156-0457-4db2-bc3c-17cc251a2076-kube-api-access-qnczq\") pod \"cert-manager-cainjector-5c57bbd4fc-t4jbt\" (UID: \"b1461156-0457-4db2-bc3c-17cc251a2076\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.267306 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.267276 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1461156-0457-4db2-bc3c-17cc251a2076-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t4jbt\" (UID: \"b1461156-0457-4db2-bc3c-17cc251a2076\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.384261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.384166 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" Nov 03 03:09:10.421637 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.421600 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" event={"ID":"c3bb8fd4-ba56-48e2-a299-a5592943018b","Type":"ContainerStarted","Data":"c4925bf1060b72a6928ec312a2dad65d47586350323324989d17448d9acfe9b2"} Nov 03 03:09:10.421787 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.421715 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:10.553307 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.553244 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" podStartSLOduration=2.03500799 podStartE2EDuration="6.553227595s" totalImagesPullingTime="4.518219605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:10.550712213 +0000 UTC m=+485.581310750" watchObservedRunningTime="2026-11-03 03:09:10.553227595 +0000 UTC m=+485.583826132" Nov 03 03:09:10.764297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:10.764274 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt"] Nov 03 03:09:10.766054 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:09:10.766018 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1461156_0457_4db2_bc3c_17cc251a2076.slice/crio-908369704c98ec39fd72909579e057e492904a08a5ff674ac595c9c309c8a5db WatchSource:0}: Error finding container 908369704c98ec39fd72909579e057e492904a08a5ff674ac595c9c309c8a5db: Status 404 returned error can't find the container with id 908369704c98ec39fd72909579e057e492904a08a5ff674ac595c9c309c8a5db Nov 03 03:09:11.428395 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:11.428353 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" event={"ID":"b1461156-0457-4db2-bc3c-17cc251a2076","Type":"ContainerStarted","Data":"3db10d0b0144b55bdf23cfe0facb6feedae575c8a9a5df3ea0a147e812eb0839"} Nov 03 03:09:11.428892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:11.428412 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" event={"ID":"b1461156-0457-4db2-bc3c-17cc251a2076","Type":"ContainerStarted","Data":"908369704c98ec39fd72909579e057e492904a08a5ff674ac595c9c309c8a5db"} Nov 03 03:09:11.573744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:11.573683 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t4jbt" podStartSLOduration=1.573669632 podStartE2EDuration="1.573669632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:11.5727433 +0000 UTC m=+486.603341839" watchObservedRunningTime="2026-11-03 03:09:11.573669632 +0000 UTC m=+486.604268169" Nov 03 03:09:14.121259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.121214 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw"] Nov 03 03:09:14.124781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.124762 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.127545 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.127512 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6hv\" (UniqueName: \"kubernetes.io/projected/dfba0d42-1654-48c7-b958-e496bb4b94a3-kube-api-access-pc6hv\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.127680 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.127576 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfba0d42-1654-48c7-b958-e496bb4b94a3-tls-certs\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.127680 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.127634 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfba0d42-1654-48c7-b958-e496bb4b94a3-tmp\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.196519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.196482 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 03:09:14.196519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.196487 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jgrrk\"" Nov 03 03:09:14.228303 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.228272 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6hv\" (UniqueName: \"kubernetes.io/projected/dfba0d42-1654-48c7-b958-e496bb4b94a3-kube-api-access-pc6hv\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.228497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.228316 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfba0d42-1654-48c7-b958-e496bb4b94a3-tls-certs\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.228497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.228333 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfba0d42-1654-48c7-b958-e496bb4b94a3-tmp\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.230673 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.230646 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfba0d42-1654-48c7-b958-e496bb4b94a3-tmp\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.230916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.230898 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfba0d42-1654-48c7-b958-e496bb4b94a3-tls-certs\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.247255 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.247218 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 03:09:14.249091 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.249066 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw"] Nov 03 03:09:14.318285 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.318254 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6hv\" (UniqueName: \"kubernetes.io/projected/dfba0d42-1654-48c7-b958-e496bb4b94a3-kube-api-access-pc6hv\") pod \"kube-auth-proxy-d5cd8cf48-2vqdw\" (UID: \"dfba0d42-1654-48c7-b958-e496bb4b94a3\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.438776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.438697 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" Nov 03 03:09:14.838295 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:14.838272 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw"] Nov 03 03:09:14.840347 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:09:14.840319 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfba0d42_1654_48c7_b958_e496bb4b94a3.slice/crio-31c5913d51a319f6ede027f9a7772181ea149ab4c4b072d688d3ce5dbc3ec44f WatchSource:0}: Error finding container 31c5913d51a319f6ede027f9a7772181ea149ab4c4b072d688d3ce5dbc3ec44f: Status 404 returned error can't find the container with id 31c5913d51a319f6ede027f9a7772181ea149ab4c4b072d688d3ce5dbc3ec44f Nov 03 03:09:15.450651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:15.450617 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" event={"ID":"dfba0d42-1654-48c7-b958-e496bb4b94a3","Type":"ContainerStarted","Data":"31c5913d51a319f6ede027f9a7772181ea149ab4c4b072d688d3ce5dbc3ec44f"} Nov 03 03:09:16.431328 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:16.431273 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-kvtg9" Nov 03 03:09:16.754999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:16.754920 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cdhfj"] Nov 03 03:09:16.758720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:16.758698 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" Nov 03 03:09:16.829424 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:16.829386 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cdhfj"] Nov 03 03:09:16.844858 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:16.844828 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-84rtq\"" Nov 03 03:09:16.849372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:16.849337 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/d9d4dc69-e045-4514-863d-f5998dba6c9e-kube-api-access-pvlzn\") pod \"authorino-57f7f8659f-cdhfj\" (UID: \"d9d4dc69-e045-4514-863d-f5998dba6c9e\") " pod="kuadrant-system/authorino-57f7f8659f-cdhfj" Nov 03 03:09:16.949857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:16.949821 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/d9d4dc69-e045-4514-863d-f5998dba6c9e-kube-api-access-pvlzn\") pod \"authorino-57f7f8659f-cdhfj\" (UID: \"d9d4dc69-e045-4514-863d-f5998dba6c9e\") " pod="kuadrant-system/authorino-57f7f8659f-cdhfj" Nov 03 03:09:17.020681 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:17.020608 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/d9d4dc69-e045-4514-863d-f5998dba6c9e-kube-api-access-pvlzn\") pod \"authorino-57f7f8659f-cdhfj\" (UID: \"d9d4dc69-e045-4514-863d-f5998dba6c9e\") " pod="kuadrant-system/authorino-57f7f8659f-cdhfj" Nov 03 03:09:17.073601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:17.073565 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" Nov 03 03:09:17.471966 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:17.471941 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cdhfj"] Nov 03 03:09:17.474107 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:09:17.474078 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9d4dc69_e045_4514_863d_f5998dba6c9e.slice/crio-71f09a92ee04c903e8725b90de7495a6cd8683f43ad82e4c930a706a2bbd1efe WatchSource:0}: Error finding container 71f09a92ee04c903e8725b90de7495a6cd8683f43ad82e4c930a706a2bbd1efe: Status 404 returned error can't find the container with id 71f09a92ee04c903e8725b90de7495a6cd8683f43ad82e4c930a706a2bbd1efe Nov 03 03:09:18.468961 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:18.468926 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" event={"ID":"dfba0d42-1654-48c7-b958-e496bb4b94a3","Type":"ContainerStarted","Data":"7f3a66dc6303acd13c0e590ef051e7069965c5b9f4582eae3cafaba6fb92ff5e"} Nov 03 03:09:18.470226 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:18.470199 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" event={"ID":"d9d4dc69-e045-4514-863d-f5998dba6c9e","Type":"ContainerStarted","Data":"71f09a92ee04c903e8725b90de7495a6cd8683f43ad82e4c930a706a2bbd1efe"} Nov 03 03:09:18.597122 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:18.597064 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-2vqdw" podStartSLOduration=1.4406607519999999 podStartE2EDuration="4.597047118s" totalImagesPullingTime="3.156386366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:18.595771553 +0000 UTC m=+493.626370091" watchObservedRunningTime="2026-11-03 03:09:18.597047118 +0000 UTC m=+493.627645655" Nov 03 03:09:22.491407 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:22.491316 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" event={"ID":"d9d4dc69-e045-4514-863d-f5998dba6c9e","Type":"ContainerStarted","Data":"63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5"} Nov 03 03:09:22.611529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:22.611476 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" podStartSLOduration=1.995234535 podStartE2EDuration="6.611461517s" totalImagesPullingTime="4.616226982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:22.61027291 +0000 UTC m=+497.640871449" watchObservedRunningTime="2026-11-03 03:09:22.611461517 +0000 UTC m=+497.642060046" Nov 03 03:09:23.205713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:23.205670 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cdhfj"] Nov 03 03:09:24.501661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:24.501622 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" podUID="d9d4dc69-e045-4514-863d-f5998dba6c9e" containerName="authorino" containerID="cri-o://63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5" gracePeriod=30 Nov 03 03:09:24.750550 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:24.750525 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" Nov 03 03:09:24.806805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:24.806718 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d4dc69-e045-4514-863d-f5998dba6c9e-kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/d9d4dc69-e045-4514-863d-f5998dba6c9e-kube-api-access-pvlzn\") pod \"d9d4dc69-e045-4514-863d-f5998dba6c9e\" (UID: \"d9d4dc69-e045-4514-863d-f5998dba6c9e\") " Nov 03 03:09:24.809094 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:24.809063 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d4dc69-e045-4514-863d-f5998dba6c9e-kube-api-access-pvlzn" pod "d9d4dc69-e045-4514-863d-f5998dba6c9e" (UID: "d9d4dc69-e045-4514-863d-f5998dba6c9e"). InnerVolumeSpecName "kube-api-access-pvlzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:24.907747 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:24.907716 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/d9d4dc69-e045-4514-863d-f5998dba6c9e-kube-api-access-pvlzn\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:09:25.507713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.507617 5117 generic.go:356] "Generic (PLEG): container finished" podID="d9d4dc69-e045-4514-863d-f5998dba6c9e" containerID="63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5" exitCode=0 Nov 03 03:09:25.507713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.507655 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" event={"ID":"d9d4dc69-e045-4514-863d-f5998dba6c9e","Type":"ContainerDied","Data":"63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5"} Nov 03 03:09:25.507713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.507669 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" Nov 03 03:09:25.507713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.507687 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cdhfj" event={"ID":"d9d4dc69-e045-4514-863d-f5998dba6c9e","Type":"ContainerDied","Data":"71f09a92ee04c903e8725b90de7495a6cd8683f43ad82e4c930a706a2bbd1efe"} Nov 03 03:09:25.507713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.507704 5117 scope.go:122] "RemoveContainer" containerID="63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5" Nov 03 03:09:25.516731 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.516713 5117 scope.go:122] "RemoveContainer" containerID="63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5" Nov 03 03:09:25.517019 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:09:25.517000 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5\": container with ID starting with 63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5 not found: ID does not exist" containerID="63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5" Nov 03 03:09:25.517057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.517030 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5"} err="failed to get container status \"63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5\": rpc error: code = NotFound desc = could not find container \"63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5\": container with ID starting with 63eb0145fda99d821880528b125227370bae31281869439a759088f02cbb65c5 not found: ID does not exist" Nov 03 03:09:25.675722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.675681 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cdhfj"] Nov 03 03:09:25.684752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:25.684721 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cdhfj"] Nov 03 03:09:27.088504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:27.088470 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d4dc69-e045-4514-863d-f5998dba6c9e" path="/var/lib/kubelet/pods/d9d4dc69-e045-4514-863d-f5998dba6c9e/volumes" Nov 03 03:09:30.010339 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:09:30.010297 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:09:40.761709 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.761674 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx"] Nov 03 03:09:40.762212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.762024 5117 cpu_manager.go:414] "containerMap: removing container" podUID="d9d4dc69-e045-4514-863d-f5998dba6c9e" containerName="authorino" Nov 03 03:09:40.762212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.762035 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d4dc69-e045-4514-863d-f5998dba6c9e" containerName="authorino" Nov 03 03:09:40.762212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.762118 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d4dc69-e045-4514-863d-f5998dba6c9e" containerName="authorino" Nov 03 03:09:40.763798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.763783 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:40.818105 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.818073 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 03:09:40.822489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.822459 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/568412cb-2ac5-4a39-a98f-92248dbfc164-data\") pod \"postgres-7cdbf6dc44-t94jx\" (UID: \"568412cb-2ac5-4a39-a98f-92248dbfc164\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:40.822619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.822516 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnn5v\" (UniqueName: \"kubernetes.io/projected/568412cb-2ac5-4a39-a98f-92248dbfc164-kube-api-access-hnn5v\") pod \"postgres-7cdbf6dc44-t94jx\" (UID: \"568412cb-2ac5-4a39-a98f-92248dbfc164\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:40.868516 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.868478 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:09:40.868516 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.868478 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-g84jj\"" Nov 03 03:09:40.868739 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.868478 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:09:40.870559 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.870539 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx"] Nov 03 03:09:40.923022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.922987 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/568412cb-2ac5-4a39-a98f-92248dbfc164-data\") pod \"postgres-7cdbf6dc44-t94jx\" (UID: \"568412cb-2ac5-4a39-a98f-92248dbfc164\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:40.923214 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.923047 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnn5v\" (UniqueName: \"kubernetes.io/projected/568412cb-2ac5-4a39-a98f-92248dbfc164-kube-api-access-hnn5v\") pod \"postgres-7cdbf6dc44-t94jx\" (UID: \"568412cb-2ac5-4a39-a98f-92248dbfc164\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:40.923507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.923486 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/568412cb-2ac5-4a39-a98f-92248dbfc164-data\") pod \"postgres-7cdbf6dc44-t94jx\" (UID: \"568412cb-2ac5-4a39-a98f-92248dbfc164\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:40.991574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:40.991535 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnn5v\" (UniqueName: \"kubernetes.io/projected/568412cb-2ac5-4a39-a98f-92248dbfc164-kube-api-access-hnn5v\") pod \"postgres-7cdbf6dc44-t94jx\" (UID: \"568412cb-2ac5-4a39-a98f-92248dbfc164\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:41.075409 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:41.075367 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:41.449523 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:41.449435 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx"] Nov 03 03:09:41.452040 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:09:41.452011 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568412cb_2ac5_4a39_a98f_92248dbfc164.slice/crio-a5de1a73dfb733206595a8a57b1abcf4f3965d49b96ec8e68c82585c08305461 WatchSource:0}: Error finding container a5de1a73dfb733206595a8a57b1abcf4f3965d49b96ec8e68c82585c08305461: Status 404 returned error can't find the container with id a5de1a73dfb733206595a8a57b1abcf4f3965d49b96ec8e68c82585c08305461 Nov 03 03:09:41.593631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:41.593595 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" event={"ID":"568412cb-2ac5-4a39-a98f-92248dbfc164","Type":"ContainerStarted","Data":"a5de1a73dfb733206595a8a57b1abcf4f3965d49b96ec8e68c82585c08305461"} Nov 03 03:09:48.631801 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:48.631758 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" event={"ID":"568412cb-2ac5-4a39-a98f-92248dbfc164","Type":"ContainerStarted","Data":"5f72bf0e4de7d4e77fd85abd6709ffeb5a9bfd0a4aeb5ebf5836355f9c2707fc"} Nov 03 03:09:48.632277 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:48.631906 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:09:48.752173 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:48.752117 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" podStartSLOduration=2.447731433 podStartE2EDuration="8.752101504s" totalImagesPullingTime="6.304370071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:48.750519206 +0000 UTC m=+523.781117744" watchObservedRunningTime="2026-11-03 03:09:48.752101504 +0000 UTC m=+523.782700042" Nov 03 03:09:54.667255 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:09:54.667224 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-t94jx" Nov 03 03:10:00.009824 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:10:00.009780 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:10:02.810463 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.810412 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66c9557c77-2cp7g"] Nov 03 03:10:02.818831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.818804 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:02.872892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.872859 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 03:10:02.873066 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.872906 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 03:10:02.894994 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.894960 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 03:10:02.894994 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.894989 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 03:10:02.921372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.921338 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-webhook-cert\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:02.921568 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.921379 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-metrics-certs\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:02.921640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.921583 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxq4\" (UniqueName: \"kubernetes.io/projected/77955383-4ced-456b-ab9d-b886daf8e40e-kube-api-access-8qxq4\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:02.923419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.923399 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4f44h\"" Nov 03 03:10:02.923529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.923403 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 03:10:02.925352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:02.925335 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66c9557c77-2cp7g"] Nov 03 03:10:03.022470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.022405 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-metrics-certs\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:03.022665 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.022530 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxq4\" (UniqueName: \"kubernetes.io/projected/77955383-4ced-456b-ab9d-b886daf8e40e-kube-api-access-8qxq4\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:03.022665 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.022579 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-webhook-cert\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:03.025309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.025285 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-webhook-cert\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:03.025455 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.025423 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-metrics-certs\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:03.086124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.086045 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxq4\" (UniqueName: \"kubernetes.io/projected/77955383-4ced-456b-ab9d-b886daf8e40e-kube-api-access-8qxq4\") pod \"maas-controller-66c9557c77-2cp7g\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:03.132351 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.132307 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:03.478669 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.478589 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7957588c98-fsqfj"] Nov 03 03:10:03.483736 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.483714 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.543356 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.543326 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7957588c98-fsqfj"] Nov 03 03:10:03.558539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.558515 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66c9557c77-2cp7g"] Nov 03 03:10:03.560367 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:10:03.560324 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77955383_4ced_456b_ab9d_b886daf8e40e.slice/crio-746e104d17b31c3075ce677d76370ab95c080cb81c4ab3208a03f1ada5968df6 WatchSource:0}: Error finding container 746e104d17b31c3075ce677d76370ab95c080cb81c4ab3208a03f1ada5968df6: Status 404 returned error can't find the container with id 746e104d17b31c3075ce677d76370ab95c080cb81c4ab3208a03f1ada5968df6 Nov 03 03:10:03.626817 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.626774 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-webhook-cert\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.626817 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.626822 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzhm\" (UniqueName: \"kubernetes.io/projected/4c3696fd-b5a8-4c37-ad9b-888a384beb12-kube-api-access-4pzhm\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.627034 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.626954 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-metrics-certs\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.701727 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.701683 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-2cp7g" event={"ID":"77955383-4ced-456b-ab9d-b886daf8e40e","Type":"ContainerStarted","Data":"746e104d17b31c3075ce677d76370ab95c080cb81c4ab3208a03f1ada5968df6"} Nov 03 03:10:03.728184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.728148 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-webhook-cert\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.728357 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.728189 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzhm\" (UniqueName: \"kubernetes.io/projected/4c3696fd-b5a8-4c37-ad9b-888a384beb12-kube-api-access-4pzhm\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.728357 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.728240 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-metrics-certs\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.730987 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.730927 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-metrics-certs\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.730987 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.730927 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-webhook-cert\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.791595 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.791561 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzhm\" (UniqueName: \"kubernetes.io/projected/4c3696fd-b5a8-4c37-ad9b-888a384beb12-kube-api-access-4pzhm\") pod \"maas-controller-7957588c98-fsqfj\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:03.797423 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:03.797398 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:04.174819 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:04.174788 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7957588c98-fsqfj"] Nov 03 03:10:04.177090 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:10:04.177062 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3696fd_b5a8_4c37_ad9b_888a384beb12.slice/crio-58e9d9ed4171807a5ae67060df22f7901b61921bcda418d93c41b5345562df7d WatchSource:0}: Error finding container 58e9d9ed4171807a5ae67060df22f7901b61921bcda418d93c41b5345562df7d: Status 404 returned error can't find the container with id 58e9d9ed4171807a5ae67060df22f7901b61921bcda418d93c41b5345562df7d Nov 03 03:10:04.708152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:04.708119 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-fsqfj" event={"ID":"4c3696fd-b5a8-4c37-ad9b-888a384beb12","Type":"ContainerStarted","Data":"58e9d9ed4171807a5ae67060df22f7901b61921bcda418d93c41b5345562df7d"} Nov 03 03:10:09.265101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:09.265065 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 03:10:09.736950 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:09.736837 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-2cp7g" event={"ID":"77955383-4ced-456b-ab9d-b886daf8e40e","Type":"ContainerStarted","Data":"eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75"} Nov 03 03:10:09.736950 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:09.736913 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:09.738628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:09.738591 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-fsqfj" event={"ID":"4c3696fd-b5a8-4c37-ad9b-888a384beb12","Type":"ContainerStarted","Data":"306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14"} Nov 03 03:10:09.738739 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:09.738728 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:09.860123 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:09.860070 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66c9557c77-2cp7g" podStartSLOduration=2.210679203 podStartE2EDuration="7.860053632s" totalImagesPullingTime="5.649374429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:09.858983448 +0000 UTC m=+544.889581986" watchObservedRunningTime="2026-11-03 03:10:09.860053632 +0000 UTC m=+544.890652172" Nov 03 03:10:09.981117 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:09.981067 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7957588c98-fsqfj" podStartSLOduration=1.941591611 podStartE2EDuration="6.981053666s" totalImagesPullingTime="5.039462055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:09.980781589 +0000 UTC m=+545.011380126" watchObservedRunningTime="2026-11-03 03:10:09.981053666 +0000 UTC m=+545.011652204" Nov 03 03:10:12.739140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.739093 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf"] Nov 03 03:10:12.743040 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.743013 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:12.796857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.796820 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 03:10:12.796857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.796820 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 03:10:12.796857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.796836 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 03:10:12.847917 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.847886 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-sgzkg\"" Nov 03 03:10:12.850990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.850963 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf"] Nov 03 03:10:12.914098 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.914055 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:12.914098 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.914102 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx4f9\" (UniqueName: \"kubernetes.io/projected/9699142d-1bf4-46a4-9705-f5bd4c028ae6-kube-api-access-cx4f9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:12.914304 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.914252 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9699142d-1bf4-46a4-9705-f5bd4c028ae6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:12.914304 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:12.914287 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.015938 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.015779 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9699142d-1bf4-46a4-9705-f5bd4c028ae6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.015938 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.015823 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.015938 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.015878 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.015938 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.015920 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx4f9\" (UniqueName: \"kubernetes.io/projected/9699142d-1bf4-46a4-9705-f5bd4c028ae6-kube-api-access-cx4f9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.016572 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:10:13.016012 5117 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 03:10:13.016572 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:10:13.016147 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-proxy-source podName:9699142d-1bf4-46a4-9705-f5bd4c028ae6 nodeName:}" failed. No retries permitted until 2026-11-03 03:10:13.516125066 +0000 UTC m=+548.546723582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" (UID: "9699142d-1bf4-46a4-9705-f5bd4c028ae6") : configmap "usage-tenancy-proxy-source" not found Nov 03 03:10:13.016921 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.016873 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.019371 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.019353 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9699142d-1bf4-46a4-9705-f5bd4c028ae6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.080751 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.080717 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx4f9\" (UniqueName: \"kubernetes.io/projected/9699142d-1bf4-46a4-9705-f5bd4c028ae6-kube-api-access-cx4f9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.521436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.521396 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.522285 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.522263 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9699142d-1bf4-46a4-9705-f5bd4c028ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-pzvwf\" (UID: \"9699142d-1bf4-46a4-9705-f5bd4c028ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:13.657093 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:13.657033 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:14.040714 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:14.040685 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf"] Nov 03 03:10:14.042123 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:10:14.042096 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9699142d_1bf4_46a4_9705_f5bd4c028ae6.slice/crio-41e02d60b96ba508b7371338f0babd7470e663702266a53fe7173d824593f37d WatchSource:0}: Error finding container 41e02d60b96ba508b7371338f0babd7470e663702266a53fe7173d824593f37d: Status 404 returned error can't find the container with id 41e02d60b96ba508b7371338f0babd7470e663702266a53fe7173d824593f37d Nov 03 03:10:14.763686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:14.763647 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" event={"ID":"9699142d-1bf4-46a4-9705-f5bd4c028ae6","Type":"ContainerStarted","Data":"41e02d60b96ba508b7371338f0babd7470e663702266a53fe7173d824593f37d"} Nov 03 03:10:20.745967 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:20.745931 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:10:20.746424 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:20.746021 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:20.921624 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:20.921176 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66c9557c77-2cp7g"] Nov 03 03:10:20.921624 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:20.921426 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66c9557c77-2cp7g" podUID="77955383-4ced-456b-ab9d-b886daf8e40e" containerName="manager" containerID="cri-o://eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75" gracePeriod=30 Nov 03 03:10:21.694923 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.694900 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:21.797770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.797671 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-metrics-certs\") pod \"77955383-4ced-456b-ab9d-b886daf8e40e\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " Nov 03 03:10:21.797770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.797757 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77955383-4ced-456b-ab9d-b886daf8e40e-kube-api-access-8qxq4\" (UniqueName: \"kubernetes.io/projected/77955383-4ced-456b-ab9d-b886daf8e40e-kube-api-access-8qxq4\") pod \"77955383-4ced-456b-ab9d-b886daf8e40e\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " Nov 03 03:10:21.798329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.797843 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-webhook-cert\") pod \"77955383-4ced-456b-ab9d-b886daf8e40e\" (UID: \"77955383-4ced-456b-ab9d-b886daf8e40e\") " Nov 03 03:10:21.800397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.800364 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-webhook-cert" pod "77955383-4ced-456b-ab9d-b886daf8e40e" (UID: "77955383-4ced-456b-ab9d-b886daf8e40e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:21.800542 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.800365 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77955383-4ced-456b-ab9d-b886daf8e40e-kube-api-access-8qxq4" pod "77955383-4ced-456b-ab9d-b886daf8e40e" (UID: "77955383-4ced-456b-ab9d-b886daf8e40e"). InnerVolumeSpecName "kube-api-access-8qxq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:21.800542 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.800500 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-metrics-certs" pod "77955383-4ced-456b-ab9d-b886daf8e40e" (UID: "77955383-4ced-456b-ab9d-b886daf8e40e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:21.806751 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.806710 5117 generic.go:356] "Generic (PLEG): container finished" podID="77955383-4ced-456b-ab9d-b886daf8e40e" containerID="eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75" exitCode=0 Nov 03 03:10:21.806929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.806809 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c9557c77-2cp7g" Nov 03 03:10:21.806929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.806825 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-2cp7g" event={"ID":"77955383-4ced-456b-ab9d-b886daf8e40e","Type":"ContainerDied","Data":"eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75"} Nov 03 03:10:21.806929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.806909 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c9557c77-2cp7g" event={"ID":"77955383-4ced-456b-ab9d-b886daf8e40e","Type":"ContainerDied","Data":"746e104d17b31c3075ce677d76370ab95c080cb81c4ab3208a03f1ada5968df6"} Nov 03 03:10:21.807114 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.806940 5117 scope.go:122] "RemoveContainer" containerID="eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75" Nov 03 03:10:21.824858 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.824828 5117 scope.go:122] "RemoveContainer" containerID="eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75" Nov 03 03:10:21.825394 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:10:21.825366 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75\": container with ID starting with eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75 not found: ID does not exist" containerID="eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75" Nov 03 03:10:21.825537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.825407 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75"} err="failed to get container status \"eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75\": rpc error: code = NotFound desc = could not find container \"eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75\": container with ID starting with eaba045c83ef46502889f6b25860a8768eb27b46a1bf9d5f0908d1fec6c80c75 not found: ID does not exist" Nov 03 03:10:21.898911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.898872 5117 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-webhook-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:10:21.898911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.898903 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77955383-4ced-456b-ab9d-b886daf8e40e-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:10:21.898911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.898916 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxq4\" (UniqueName: \"kubernetes.io/projected/77955383-4ced-456b-ab9d-b886daf8e40e-kube-api-access-8qxq4\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:10:21.986495 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.986435 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66c9557c77-2cp7g"] Nov 03 03:10:21.990831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:21.990790 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66c9557c77-2cp7g"] Nov 03 03:10:23.090014 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:23.089970 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77955383-4ced-456b-ab9d-b886daf8e40e" path="/var/lib/kubelet/pods/77955383-4ced-456b-ab9d-b886daf8e40e/volumes" Nov 03 03:10:30.020483 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:10:30.020348 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:10:30.860361 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:30.860316 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" event={"ID":"9699142d-1bf4-46a4-9705-f5bd4c028ae6","Type":"ContainerStarted","Data":"1383c1ac96de227a83396864920cd8517cebcaca79c50b308704b69c38475b60"} Nov 03 03:10:30.860575 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:30.860468 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:30.981385 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:30.981330 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" podStartSLOduration=2.540278828 podStartE2EDuration="18.981316596s" totalImagesPullingTime="16.441037768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:30.980665016 +0000 UTC m=+566.011263558" watchObservedRunningTime="2026-11-03 03:10:30.981316596 +0000 UTC m=+566.011915167" Nov 03 03:10:41.871831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:41.871744 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-pzvwf" Nov 03 03:10:44.120211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.120174 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw"] Nov 03 03:10:44.120770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.120619 5117 cpu_manager.go:414] "containerMap: removing container" podUID="77955383-4ced-456b-ab9d-b886daf8e40e" containerName="manager" Nov 03 03:10:44.120770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.120632 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77955383-4ced-456b-ab9d-b886daf8e40e" containerName="manager" Nov 03 03:10:44.120770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.120692 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="77955383-4ced-456b-ab9d-b886daf8e40e" containerName="manager" Nov 03 03:10:44.127119 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.127096 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.195007 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.194967 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-maas-api-tls\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.195175 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.195020 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-metrics-certs\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.195175 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.195046 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x29z\" (UniqueName: \"kubernetes.io/projected/5238c988-7b5c-43e5-bc34-ddfa803fcda4-kube-api-access-2x29z\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.195245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.195184 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5238c988-7b5c-43e5-bc34-ddfa803fcda4-openshift-service-ca\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.232595 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.232557 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcxk\"" Nov 03 03:10:44.232765 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.232625 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 03:10:44.232765 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.232723 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 03:10:44.234653 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.234631 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw"] Nov 03 03:10:44.295798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.295762 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-maas-api-tls\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.296008 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.295808 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-metrics-certs\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.296008 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.295834 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x29z\" (UniqueName: \"kubernetes.io/projected/5238c988-7b5c-43e5-bc34-ddfa803fcda4-kube-api-access-2x29z\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.296008 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.295856 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5238c988-7b5c-43e5-bc34-ddfa803fcda4-openshift-service-ca\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.296576 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.296522 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5238c988-7b5c-43e5-bc34-ddfa803fcda4-openshift-service-ca\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.298470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.298404 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-metrics-certs\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.298566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.298503 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-maas-api-tls\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.360329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.360299 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x29z\" (UniqueName: \"kubernetes.io/projected/5238c988-7b5c-43e5-bc34-ddfa803fcda4-kube-api-access-2x29z\") pod \"maas-api-5ccf48f65-c4mlw\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.440630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.440531 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:44.821828 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.821804 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw"] Nov 03 03:10:44.823544 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:10:44.823515 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5238c988_7b5c_43e5_bc34_ddfa803fcda4.slice/crio-719623e7d5f3155702cc659b5049e38610b5177c8cb65f6f531f7c50f5b710e6 WatchSource:0}: Error finding container 719623e7d5f3155702cc659b5049e38610b5177c8cb65f6f531f7c50f5b710e6: Status 404 returned error can't find the container with id 719623e7d5f3155702cc659b5049e38610b5177c8cb65f6f531f7c50f5b710e6 Nov 03 03:10:44.930350 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:44.930314 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" event={"ID":"5238c988-7b5c-43e5-bc34-ddfa803fcda4","Type":"ContainerStarted","Data":"719623e7d5f3155702cc659b5049e38610b5177c8cb65f6f531f7c50f5b710e6"} Nov 03 03:10:48.952087 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:48.951993 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" event={"ID":"5238c988-7b5c-43e5-bc34-ddfa803fcda4","Type":"ContainerStarted","Data":"794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38"} Nov 03 03:10:48.952533 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:48.952153 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:49.072348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:49.072293 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" podStartSLOduration=1.269599591 podStartE2EDuration="5.072279807s" totalImagesPullingTime="3.802680216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:49.071974781 +0000 UTC m=+584.102573322" watchObservedRunningTime="2026-11-03 03:10:49.072279807 +0000 UTC m=+584.102878386" Nov 03 03:10:50.861558 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:50.861512 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd"] Nov 03 03:10:50.865266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:50.865247 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:50.949093 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:50.949052 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c249b25-205c-4fd3-a4d0-92d9bea3d890-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r5vdd\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:50.949299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:50.949110 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8pf\" (UniqueName: \"kubernetes.io/projected/5c249b25-205c-4fd3-a4d0-92d9bea3d890-kube-api-access-hc8pf\") pod \"llmisvc-controller-manager-869d5dcc5d-r5vdd\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:50.958176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:50.958136 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd"] Nov 03 03:10:50.964359 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:50.964334 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v4qz7\"" Nov 03 03:10:50.964651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:50.964635 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 03:10:51.050088 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.050050 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c249b25-205c-4fd3-a4d0-92d9bea3d890-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r5vdd\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:51.050254 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.050108 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8pf\" (UniqueName: \"kubernetes.io/projected/5c249b25-205c-4fd3-a4d0-92d9bea3d890-kube-api-access-hc8pf\") pod \"llmisvc-controller-manager-869d5dcc5d-r5vdd\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:51.053028 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.052998 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c249b25-205c-4fd3-a4d0-92d9bea3d890-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r5vdd\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:51.110148 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.110118 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-mn4ss"] Nov 03 03:10:51.113660 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.113610 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.118548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.118521 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8pf\" (UniqueName: \"kubernetes.io/projected/5c249b25-205c-4fd3-a4d0-92d9bea3d890-kube-api-access-hc8pf\") pod \"llmisvc-controller-manager-869d5dcc5d-r5vdd\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:51.150995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.150968 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/636bbe26-5388-49da-93d3-7d53a66b7d94-cert\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.151182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.151010 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636bbe26-5388-49da-93d3-7d53a66b7d94-metrics-certs\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.151182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.151112 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6h54\" (UniqueName: \"kubernetes.io/projected/636bbe26-5388-49da-93d3-7d53a66b7d94-kube-api-access-z6h54\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.170681 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.170657 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pwpqv\"" Nov 03 03:10:51.170681 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.170672 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 03:10:51.170899 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.170702 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 03:10:51.173255 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.173229 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mn4ss"] Nov 03 03:10:51.177872 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.177852 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:51.252534 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.252499 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6h54\" (UniqueName: \"kubernetes.io/projected/636bbe26-5388-49da-93d3-7d53a66b7d94-kube-api-access-z6h54\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.252692 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.252595 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/636bbe26-5388-49da-93d3-7d53a66b7d94-cert\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.252692 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.252632 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636bbe26-5388-49da-93d3-7d53a66b7d94-metrics-certs\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.255270 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.255227 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/636bbe26-5388-49da-93d3-7d53a66b7d94-cert\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.255392 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.255359 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/636bbe26-5388-49da-93d3-7d53a66b7d94-metrics-certs\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.314678 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.314643 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6h54\" (UniqueName: \"kubernetes.io/projected/636bbe26-5388-49da-93d3-7d53a66b7d94-kube-api-access-z6h54\") pod \"odh-model-controller-769875db56-mn4ss\" (UID: \"636bbe26-5388-49da-93d3-7d53a66b7d94\") " pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.434433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.434344 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:51.555123 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.555093 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd"] Nov 03 03:10:51.557304 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:10:51.557281 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c249b25_205c_4fd3_a4d0_92d9bea3d890.slice/crio-2938682524a053b4bcedfd18ee714d3f72ad97f6a1c7c5e30da1daaa9e67a54a WatchSource:0}: Error finding container 2938682524a053b4bcedfd18ee714d3f72ad97f6a1c7c5e30da1daaa9e67a54a: Status 404 returned error can't find the container with id 2938682524a053b4bcedfd18ee714d3f72ad97f6a1c7c5e30da1daaa9e67a54a Nov 03 03:10:51.810533 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.810510 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mn4ss"] Nov 03 03:10:51.811862 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:10:51.811839 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636bbe26_5388_49da_93d3_7d53a66b7d94.slice/crio-2d27bce3133c3f0c619f91f73e5bda4c8daf1de35b16dec555da432abbe37f5f WatchSource:0}: Error finding container 2d27bce3133c3f0c619f91f73e5bda4c8daf1de35b16dec555da432abbe37f5f: Status 404 returned error can't find the container with id 2d27bce3133c3f0c619f91f73e5bda4c8daf1de35b16dec555da432abbe37f5f Nov 03 03:10:51.972503 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.972461 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mn4ss" event={"ID":"636bbe26-5388-49da-93d3-7d53a66b7d94","Type":"ContainerStarted","Data":"2d27bce3133c3f0c619f91f73e5bda4c8daf1de35b16dec555da432abbe37f5f"} Nov 03 03:10:51.973831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:51.973800 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" event={"ID":"5c249b25-205c-4fd3-a4d0-92d9bea3d890","Type":"ContainerStarted","Data":"2938682524a053b4bcedfd18ee714d3f72ad97f6a1c7c5e30da1daaa9e67a54a"} Nov 03 03:10:54.962847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:54.962809 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:10:55.979643 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:55.979559 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-77zkq"] Nov 03 03:10:55.983307 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:55.983286 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" Nov 03 03:10:55.999467 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:55.999416 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mn4ss" event={"ID":"636bbe26-5388-49da-93d3-7d53a66b7d94","Type":"ContainerStarted","Data":"340b57ca95d6237020457c23646fdbfee6532d5f84e93fe22f9876558e1cb919"} Nov 03 03:10:55.999783 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:55.999517 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:10:56.001127 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.001102 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" event={"ID":"5c249b25-205c-4fd3-a4d0-92d9bea3d890","Type":"ContainerStarted","Data":"9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623"} Nov 03 03:10:56.001226 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.001175 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:10:56.041710 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.041670 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-77zkq"] Nov 03 03:10:56.041922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.041905 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-84rtq\"" Nov 03 03:10:56.100271 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.100233 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmbg\" (UniqueName: \"kubernetes.io/projected/070ed9ac-fc00-4ae0-b89a-ac157df816ef-kube-api-access-qvmbg\") pod \"authorino-6f6944bcdd-77zkq\" (UID: \"070ed9ac-fc00-4ae0-b89a-ac157df816ef\") " pod="kuadrant-system/authorino-6f6944bcdd-77zkq" Nov 03 03:10:56.201299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.201263 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmbg\" (UniqueName: \"kubernetes.io/projected/070ed9ac-fc00-4ae0-b89a-ac157df816ef-kube-api-access-qvmbg\") pod \"authorino-6f6944bcdd-77zkq\" (UID: \"070ed9ac-fc00-4ae0-b89a-ac157df816ef\") " pod="kuadrant-system/authorino-6f6944bcdd-77zkq" Nov 03 03:10:56.236285 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.236172 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" podStartSLOduration=2.793071017 podStartE2EDuration="6.236152993s" totalImagesPullingTime="3.443081976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:56.233885108 +0000 UTC m=+591.264483647" watchObservedRunningTime="2026-11-03 03:10:56.236152993 +0000 UTC m=+591.266751531" Nov 03 03:10:56.264598 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.264564 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmbg\" (UniqueName: \"kubernetes.io/projected/070ed9ac-fc00-4ae0-b89a-ac157df816ef-kube-api-access-qvmbg\") pod \"authorino-6f6944bcdd-77zkq\" (UID: \"070ed9ac-fc00-4ae0-b89a-ac157df816ef\") " pod="kuadrant-system/authorino-6f6944bcdd-77zkq" Nov 03 03:10:56.295376 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.295325 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" Nov 03 03:10:56.357345 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.357296 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-mn4ss" podStartSLOduration=1.5549619730000002 podStartE2EDuration="5.35727932s" totalImagesPullingTime="3.802317347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:56.356371421 +0000 UTC m=+591.386969960" watchObservedRunningTime="2026-11-03 03:10:56.35727932 +0000 UTC m=+591.387877858" Nov 03 03:10:56.367946 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.367876 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-77zkq"] Nov 03 03:10:56.672013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:56.671971 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-77zkq"] Nov 03 03:10:56.673421 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:10:56.673388 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod070ed9ac_fc00_4ae0_b89a_ac157df816ef.slice/crio-11c0f3566a3719cc2c38fce67be53b80daaf3d67ca75a184cfd460bab6d690ce WatchSource:0}: Error finding container 11c0f3566a3719cc2c38fce67be53b80daaf3d67ca75a184cfd460bab6d690ce: Status 404 returned error can't find the container with id 11c0f3566a3719cc2c38fce67be53b80daaf3d67ca75a184cfd460bab6d690ce Nov 03 03:10:57.007922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:57.007831 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" event={"ID":"070ed9ac-fc00-4ae0-b89a-ac157df816ef","Type":"ContainerStarted","Data":"11c0f3566a3719cc2c38fce67be53b80daaf3d67ca75a184cfd460bab6d690ce"} Nov 03 03:10:59.024138 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:59.024097 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" event={"ID":"070ed9ac-fc00-4ae0-b89a-ac157df816ef","Type":"ContainerStarted","Data":"05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28"} Nov 03 03:10:59.024615 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:59.024139 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" podUID="070ed9ac-fc00-4ae0-b89a-ac157df816ef" containerName="authorino" containerID="cri-o://05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28" gracePeriod=30 Nov 03 03:10:59.142064 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:59.142001 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" podStartSLOduration=2.390259875 podStartE2EDuration="4.141986158s" totalImagesPullingTime="1.751726283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:59.1407352 +0000 UTC m=+594.171333738" watchObservedRunningTime="2026-11-03 03:10:59.141986158 +0000 UTC m=+594.172584696" Nov 03 03:10:59.318170 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:59.318139 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" Nov 03 03:10:59.433386 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:59.433357 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070ed9ac-fc00-4ae0-b89a-ac157df816ef-kube-api-access-qvmbg\" (UniqueName: \"kubernetes.io/projected/070ed9ac-fc00-4ae0-b89a-ac157df816ef-kube-api-access-qvmbg\") pod \"070ed9ac-fc00-4ae0-b89a-ac157df816ef\" (UID: \"070ed9ac-fc00-4ae0-b89a-ac157df816ef\") " Nov 03 03:10:59.436381 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:59.436348 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070ed9ac-fc00-4ae0-b89a-ac157df816ef-kube-api-access-qvmbg" pod "070ed9ac-fc00-4ae0-b89a-ac157df816ef" (UID: "070ed9ac-fc00-4ae0-b89a-ac157df816ef"). InnerVolumeSpecName "kube-api-access-qvmbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:59.534169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:10:59.534071 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmbg\" (UniqueName: \"kubernetes.io/projected/070ed9ac-fc00-4ae0-b89a-ac157df816ef-kube-api-access-qvmbg\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:11:00.011721 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:11:00.011680 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:11:00.030148 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.030120 5117 generic.go:356] "Generic (PLEG): container finished" podID="070ed9ac-fc00-4ae0-b89a-ac157df816ef" containerID="05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28" exitCode=0 Nov 03 03:11:00.030574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.030161 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" event={"ID":"070ed9ac-fc00-4ae0-b89a-ac157df816ef","Type":"ContainerDied","Data":"05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28"} Nov 03 03:11:00.030574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.030172 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" Nov 03 03:11:00.030574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.030193 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-77zkq" event={"ID":"070ed9ac-fc00-4ae0-b89a-ac157df816ef","Type":"ContainerDied","Data":"11c0f3566a3719cc2c38fce67be53b80daaf3d67ca75a184cfd460bab6d690ce"} Nov 03 03:11:00.030574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.030211 5117 scope.go:122] "RemoveContainer" containerID="05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28" Nov 03 03:11:00.040875 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.040847 5117 scope.go:122] "RemoveContainer" containerID="05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28" Nov 03 03:11:00.041198 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:11:00.041179 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28\": container with ID starting with 05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28 not found: ID does not exist" containerID="05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28" Nov 03 03:11:00.041293 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.041208 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28"} err="failed to get container status \"05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28\": rpc error: code = NotFound desc = could not find container \"05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28\": container with ID starting with 05d406a7e290bea209222a02206a818d687ba068c84182ed1804fc96e0e56f28 not found: ID does not exist" Nov 03 03:11:00.202313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.202275 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-77zkq"] Nov 03 03:11:00.252887 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:00.252854 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-77zkq"] Nov 03 03:11:01.089138 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:01.089101 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070ed9ac-fc00-4ae0-b89a-ac157df816ef" path="/var/lib/kubelet/pods/070ed9ac-fc00-4ae0-b89a-ac157df816ef/volumes" Nov 03 03:11:05.136410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:05.136379 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:11:05.137012 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:05.136836 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:11:06.091034 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.091005 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw"] Nov 03 03:11:06.091238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.091188 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" podUID="5238c988-7b5c-43e5-bc34-ddfa803fcda4" containerName="maas-api" containerID="cri-o://794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38" gracePeriod=30 Nov 03 03:11:06.336292 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.336269 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:11:06.497730 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.497643 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5238c988-7b5c-43e5-bc34-ddfa803fcda4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5238c988-7b5c-43e5-bc34-ddfa803fcda4-openshift-service-ca\") pod \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " Nov 03 03:11:06.497730 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.497716 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-metrics-certs\") pod \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " Nov 03 03:11:06.497933 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.497760 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-maas-api-tls\") pod \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " Nov 03 03:11:06.497933 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.497809 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5238c988-7b5c-43e5-bc34-ddfa803fcda4-kube-api-access-2x29z\" (UniqueName: \"kubernetes.io/projected/5238c988-7b5c-43e5-bc34-ddfa803fcda4-kube-api-access-2x29z\") pod \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\" (UID: \"5238c988-7b5c-43e5-bc34-ddfa803fcda4\") " Nov 03 03:11:06.498089 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.498068 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5238c988-7b5c-43e5-bc34-ddfa803fcda4-openshift-service-ca" pod "5238c988-7b5c-43e5-bc34-ddfa803fcda4" (UID: "5238c988-7b5c-43e5-bc34-ddfa803fcda4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:06.500204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.500169 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-maas-api-tls" pod "5238c988-7b5c-43e5-bc34-ddfa803fcda4" (UID: "5238c988-7b5c-43e5-bc34-ddfa803fcda4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:06.500204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.500180 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5238c988-7b5c-43e5-bc34-ddfa803fcda4-kube-api-access-2x29z" pod "5238c988-7b5c-43e5-bc34-ddfa803fcda4" (UID: "5238c988-7b5c-43e5-bc34-ddfa803fcda4"). InnerVolumeSpecName "kube-api-access-2x29z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:06.500204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.500195 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-metrics-certs" pod "5238c988-7b5c-43e5-bc34-ddfa803fcda4" (UID: "5238c988-7b5c-43e5-bc34-ddfa803fcda4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:06.599560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.599505 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:11:06.599560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.599551 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5238c988-7b5c-43e5-bc34-ddfa803fcda4-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:11:06.599560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.599562 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x29z\" (UniqueName: \"kubernetes.io/projected/5238c988-7b5c-43e5-bc34-ddfa803fcda4-kube-api-access-2x29z\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:11:06.599560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:06.599571 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5238c988-7b5c-43e5-bc34-ddfa803fcda4-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:11:07.010626 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.010590 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-mn4ss" Nov 03 03:11:07.068847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.068816 5117 generic.go:356] "Generic (PLEG): container finished" podID="5238c988-7b5c-43e5-bc34-ddfa803fcda4" containerID="794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38" exitCode=0 Nov 03 03:11:07.069019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.068861 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" Nov 03 03:11:07.069019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.068899 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" event={"ID":"5238c988-7b5c-43e5-bc34-ddfa803fcda4","Type":"ContainerDied","Data":"794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38"} Nov 03 03:11:07.069019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.068960 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw" event={"ID":"5238c988-7b5c-43e5-bc34-ddfa803fcda4","Type":"ContainerDied","Data":"719623e7d5f3155702cc659b5049e38610b5177c8cb65f6f531f7c50f5b710e6"} Nov 03 03:11:07.069019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.068985 5117 scope.go:122] "RemoveContainer" containerID="794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38" Nov 03 03:11:07.079703 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.079683 5117 scope.go:122] "RemoveContainer" containerID="794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38" Nov 03 03:11:07.080019 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:11:07.079998 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38\": container with ID starting with 794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38 not found: ID does not exist" containerID="794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38" Nov 03 03:11:07.080068 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.080028 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38"} err="failed to get container status \"794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38\": rpc error: code = NotFound desc = could not find container \"794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38\": container with ID starting with 794b6b570ccc9a8ea891c2eb2517b9ce8f58bd30db53a95b3d9d298a18da3d38 not found: ID does not exist" Nov 03 03:11:07.244942 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.244907 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw"] Nov 03 03:11:07.248841 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:07.248813 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-c4mlw"] Nov 03 03:11:09.087768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:09.087736 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5238c988-7b5c-43e5-bc34-ddfa803fcda4" path="/var/lib/kubelet/pods/5238c988-7b5c-43e5-bc34-ddfa803fcda4/volumes" Nov 03 03:11:27.010913 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:11:27.010882 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:11:30.011313 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:11:30.011275 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:12:00.010478 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:12:00.010421 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:12:01.435181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.435148 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jnblx"] Nov 03 03:12:01.435663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.435555 5117 cpu_manager.go:414] "containerMap: removing container" podUID="5238c988-7b5c-43e5-bc34-ddfa803fcda4" containerName="maas-api" Nov 03 03:12:01.435663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.435567 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5238c988-7b5c-43e5-bc34-ddfa803fcda4" containerName="maas-api" Nov 03 03:12:01.435663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.435578 5117 cpu_manager.go:414] "containerMap: removing container" podUID="070ed9ac-fc00-4ae0-b89a-ac157df816ef" containerName="authorino" Nov 03 03:12:01.435663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.435582 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070ed9ac-fc00-4ae0-b89a-ac157df816ef" containerName="authorino" Nov 03 03:12:01.435821 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.435667 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="5238c988-7b5c-43e5-bc34-ddfa803fcda4" containerName="maas-api" Nov 03 03:12:01.435821 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.435674 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="070ed9ac-fc00-4ae0-b89a-ac157df816ef" containerName="authorino" Nov 03 03:12:01.438915 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.438893 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:01.497607 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.497577 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jnblx"] Nov 03 03:12:01.521871 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.521836 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0613f917-8742-408c-80ed-640e913c05f6-cert\") pod \"llmisvc-controller-manager-6466c554d8-jnblx\" (UID: \"0613f917-8742-408c-80ed-640e913c05f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:01.522097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.521959 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9kp\" (UniqueName: \"kubernetes.io/projected/0613f917-8742-408c-80ed-640e913c05f6-kube-api-access-6c9kp\") pod \"llmisvc-controller-manager-6466c554d8-jnblx\" (UID: \"0613f917-8742-408c-80ed-640e913c05f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:01.622746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.622708 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9kp\" (UniqueName: \"kubernetes.io/projected/0613f917-8742-408c-80ed-640e913c05f6-kube-api-access-6c9kp\") pod \"llmisvc-controller-manager-6466c554d8-jnblx\" (UID: \"0613f917-8742-408c-80ed-640e913c05f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:01.622931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.622772 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0613f917-8742-408c-80ed-640e913c05f6-cert\") pod \"llmisvc-controller-manager-6466c554d8-jnblx\" (UID: \"0613f917-8742-408c-80ed-640e913c05f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:01.625423 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.625400 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0613f917-8742-408c-80ed-640e913c05f6-cert\") pod \"llmisvc-controller-manager-6466c554d8-jnblx\" (UID: \"0613f917-8742-408c-80ed-640e913c05f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:01.685730 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.685658 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9kp\" (UniqueName: \"kubernetes.io/projected/0613f917-8742-408c-80ed-640e913c05f6-kube-api-access-6c9kp\") pod \"llmisvc-controller-manager-6466c554d8-jnblx\" (UID: \"0613f917-8742-408c-80ed-640e913c05f6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:01.751989 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:01.751953 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:02.130314 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:02.130291 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jnblx"] Nov 03 03:12:02.132123 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:12:02.132095 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0613f917_8742_408c_80ed_640e913c05f6.slice/crio-112c721492d3cf7d1cc3a9429ed45e508daa737a6a7a807ec2f5bd61b99884ee WatchSource:0}: Error finding container 112c721492d3cf7d1cc3a9429ed45e508daa737a6a7a807ec2f5bd61b99884ee: Status 404 returned error can't find the container with id 112c721492d3cf7d1cc3a9429ed45e508daa737a6a7a807ec2f5bd61b99884ee Nov 03 03:12:02.344748 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:02.344710 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" event={"ID":"0613f917-8742-408c-80ed-640e913c05f6","Type":"ContainerStarted","Data":"112c721492d3cf7d1cc3a9429ed45e508daa737a6a7a807ec2f5bd61b99884ee"} Nov 03 03:12:03.351951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:03.351910 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" event={"ID":"0613f917-8742-408c-80ed-640e913c05f6","Type":"ContainerStarted","Data":"d7aac754d6fc0831a42eaed62612c0d816633e2c46f875542ea84e731f9f589d"} Nov 03 03:12:03.352491 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:03.352040 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:03.473320 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:03.473268 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" podStartSLOduration=1.731572262 podStartE2EDuration="2.473252505s" totalImagesPullingTime="741.680243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:03.472379874 +0000 UTC m=+658.502978412" watchObservedRunningTime="2026-11-03 03:12:03.473252505 +0000 UTC m=+658.503851043" Nov 03 03:12:30.010788 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:12:30.010711 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:12:34.359923 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.359893 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jnblx" Nov 03 03:12:34.532856 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.532828 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd"] Nov 03 03:12:34.533073 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.533050 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" podUID="5c249b25-205c-4fd3-a4d0-92d9bea3d890" containerName="manager" containerID="cri-o://9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623" gracePeriod=30 Nov 03 03:12:34.788261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.788239 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:12:34.914997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.914920 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c249b25-205c-4fd3-a4d0-92d9bea3d890-kube-api-access-hc8pf\" (UniqueName: \"kubernetes.io/projected/5c249b25-205c-4fd3-a4d0-92d9bea3d890-kube-api-access-hc8pf\") pod \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " Nov 03 03:12:34.915142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.915044 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c249b25-205c-4fd3-a4d0-92d9bea3d890-cert\" (UniqueName: \"kubernetes.io/secret/5c249b25-205c-4fd3-a4d0-92d9bea3d890-cert\") pod \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\" (UID: \"5c249b25-205c-4fd3-a4d0-92d9bea3d890\") " Nov 03 03:12:34.917356 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.917320 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c249b25-205c-4fd3-a4d0-92d9bea3d890-kube-api-access-hc8pf" pod "5c249b25-205c-4fd3-a4d0-92d9bea3d890" (UID: "5c249b25-205c-4fd3-a4d0-92d9bea3d890"). InnerVolumeSpecName "kube-api-access-hc8pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:34.917507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:34.917422 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c249b25-205c-4fd3-a4d0-92d9bea3d890-cert" pod "5c249b25-205c-4fd3-a4d0-92d9bea3d890" (UID: "5c249b25-205c-4fd3-a4d0-92d9bea3d890"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:35.016328 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.016287 5117 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c249b25-205c-4fd3-a4d0-92d9bea3d890-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.016328 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.016323 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8pf\" (UniqueName: \"kubernetes.io/projected/5c249b25-205c-4fd3-a4d0-92d9bea3d890-kube-api-access-hc8pf\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:12:35.512097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.512058 5117 generic.go:356] "Generic (PLEG): container finished" podID="5c249b25-205c-4fd3-a4d0-92d9bea3d890" containerID="9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623" exitCode=0 Nov 03 03:12:35.512566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.512134 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" Nov 03 03:12:35.512566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.512146 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" event={"ID":"5c249b25-205c-4fd3-a4d0-92d9bea3d890","Type":"ContainerDied","Data":"9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623"} Nov 03 03:12:35.512566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.512220 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd" event={"ID":"5c249b25-205c-4fd3-a4d0-92d9bea3d890","Type":"ContainerDied","Data":"2938682524a053b4bcedfd18ee714d3f72ad97f6a1c7c5e30da1daaa9e67a54a"} Nov 03 03:12:35.512566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.512251 5117 scope.go:122] "RemoveContainer" containerID="9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623" Nov 03 03:12:35.521213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.521195 5117 scope.go:122] "RemoveContainer" containerID="9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623" Nov 03 03:12:35.521553 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:12:35.521531 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623\": container with ID starting with 9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623 not found: ID does not exist" containerID="9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623" Nov 03 03:12:35.521608 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.521564 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623"} err="failed to get container status \"9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623\": rpc error: code = NotFound desc = could not find container \"9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623\": container with ID starting with 9c35f7672be6e6c0d9d4daed07190008065476dfa1f7f5a12e7588e4c7809623 not found: ID does not exist" Nov 03 03:12:35.684062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.684030 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd"] Nov 03 03:12:35.734680 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:35.734645 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r5vdd"] Nov 03 03:12:37.088597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:37.088560 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c249b25-205c-4fd3-a4d0-92d9bea3d890" path="/var/lib/kubelet/pods/5c249b25-205c-4fd3-a4d0-92d9bea3d890/volumes" Nov 03 03:12:39.533556 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:39.533516 5117 generic.go:356] "Generic (PLEG): container finished" podID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerID="306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14" exitCode=1 Nov 03 03:12:39.534018 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:39.533586 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-fsqfj" event={"ID":"4c3696fd-b5a8-4c37-ad9b-888a384beb12","Type":"ContainerDied","Data":"306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14"} Nov 03 03:12:39.534018 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:39.533974 5117 scope.go:122] "RemoveContainer" containerID="306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14" Nov 03 03:12:40.541140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:40.541104 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-fsqfj" event={"ID":"4c3696fd-b5a8-4c37-ad9b-888a384beb12","Type":"ContainerStarted","Data":"fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305"} Nov 03 03:12:40.541644 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:40.541321 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:12:51.548876 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:12:51.548841 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 03:13:00.010820 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:13:00.010774 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:13:30.011224 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:13:30.011183 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:14:00.010697 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:14:00.010660 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:14:01.187080 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.187048 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55"] Nov 03 03:14:01.187529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.187459 5117 cpu_manager.go:414] "containerMap: removing container" podUID="5c249b25-205c-4fd3-a4d0-92d9bea3d890" containerName="manager" Nov 03 03:14:01.187529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.187471 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c249b25-205c-4fd3-a4d0-92d9bea3d890" containerName="manager" Nov 03 03:14:01.187617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.187564 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c249b25-205c-4fd3-a4d0-92d9bea3d890" containerName="manager" Nov 03 03:14:01.190769 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.190750 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.244536 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.244500 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 03:14:01.272165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272128 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272179 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272287 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272325 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6df6\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-kube-api-access-k6df6\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272595 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272510 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272595 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272574 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272706 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272618 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272706 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272680 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.272816 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.272790 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.298383 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.298328 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55"] Nov 03 03:14:01.334066 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.334039 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gfd7r\"" Nov 03 03:14:01.373546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373508 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373558 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373676 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373716 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373747 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373785 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373815 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373838 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.373916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.373867 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6df6\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-kube-api-access-k6df6\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.374174 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.374075 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.374174 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.374114 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.374174 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.374138 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.374336 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.374264 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.374688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.374665 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.376041 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.376023 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.376288 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.376269 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.437053 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.437015 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.437053 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.437056 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6df6\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-kube-api-access-k6df6\") pod \"maas-default-gateway-openshift-default-76f4876667-7cb55\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.503995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.503898 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:01.935918 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.935888 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55"] Nov 03 03:14:01.937115 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:14:01.937084 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc555ca6b_db4c_4b18_8a3a_1a279d78556f.slice/crio-81ddbd914eeb5ee172e704d18bedc2fa8e56238588c25f5dc1934de68f6f6062 WatchSource:0}: Error finding container 81ddbd914eeb5ee172e704d18bedc2fa8e56238588c25f5dc1934de68f6f6062: Status 404 returned error can't find the container with id 81ddbd914eeb5ee172e704d18bedc2fa8e56238588c25f5dc1934de68f6f6062 Nov 03 03:14:01.939144 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.939127 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:14:01.945380 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:01.945349 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" event={"ID":"c555ca6b-db4c-4b18-8a3a-1a279d78556f","Type":"ContainerStarted","Data":"81ddbd914eeb5ee172e704d18bedc2fa8e56238588c25f5dc1934de68f6f6062"} Nov 03 03:14:06.457726 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:06.457687 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:14:06.458143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:06.457772 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:14:06.458143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:06.457820 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:14:06.975707 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:06.975669 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" event={"ID":"c555ca6b-db4c-4b18-8a3a-1a279d78556f","Type":"ContainerStarted","Data":"b32aa9da98a5ae43f5a3a8e2faa1d370322ca7ccb26c6217314ab79cf96b7df1"} Nov 03 03:14:07.149964 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:07.149910 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podStartSLOduration=1.6324570980000002 podStartE2EDuration="6.149897208s" totalImagesPullingTime="4.51744011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:07.149065983 +0000 UTC m=+782.179664521" watchObservedRunningTime="2026-11-03 03:14:07.149897208 +0000 UTC m=+782.180495745" Nov 03 03:14:07.524666 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:07.524619 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:07.526285 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:07.526251 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 03:14:07.526437 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:07.526312 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 03:14:08.514472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:08.504925 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 03:14:08.514472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:08.504998 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 03:14:09.513473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:09.504861 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 03:14:09.513473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:09.504932 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 03:14:10.524473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:10.523873 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 03:14:10.524473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:10.523931 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 03:14:11.506940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:11.506897 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 03:14:11.507262 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:11.507229 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 03:14:12.504894 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:12.504851 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 03:14:12.505296 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:12.504971 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 03:14:13.505192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:13.505144 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 03:14:13.505677 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:13.505216 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 03:14:14.829258 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:14.829222 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:14.829753 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:14.829716 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:14.831583 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:14:14.831552 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:14:30.010758 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:14:30.010706 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:15:00.011851 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:00.011812 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:15:09.609569 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.609490 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4"] Nov 03 03:15:09.613666 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.613642 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.690385 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.690350 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-metrics-certs\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.690616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.690510 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7j7\" (UniqueName: \"kubernetes.io/projected/690681d6-aa41-4b5f-9550-c9e41f620f9a-kube-api-access-wd7j7\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.690616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.690551 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.690616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.690576 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690681d6-aa41-4b5f-9550-c9e41f620f9a-openshift-service-ca\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.721634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.721595 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b4ff4779-serving-cert\"" Nov 03 03:15:09.721828 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.721686 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcxk\"" Nov 03 03:15:09.721828 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.721731 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 03:15:09.723494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.723472 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4"] Nov 03 03:15:09.791836 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.791805 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690681d6-aa41-4b5f-9550-c9e41f620f9a-openshift-service-ca\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.792035 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.791852 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-metrics-certs\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.792035 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.791920 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7j7\" (UniqueName: \"kubernetes.io/projected/690681d6-aa41-4b5f-9550-c9e41f620f9a-kube-api-access-wd7j7\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.792035 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.791947 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.792226 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:09.792041 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-serving-cert: secret "maas-api-e2e-ait-b4ff4779-serving-cert" not found Nov 03 03:15:09.792226 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:09.792106 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls podName:690681d6-aa41-4b5f-9550-c9e41f620f9a nodeName:}" failed. No retries permitted until 2026-11-03 03:15:10.29208831 +0000 UTC m=+845.322686825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls") pod "maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" (UID: "690681d6-aa41-4b5f-9550-c9e41f620f9a") : secret "maas-api-e2e-ait-b4ff4779-serving-cert" not found Nov 03 03:15:09.792645 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.792623 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690681d6-aa41-4b5f-9550-c9e41f620f9a-openshift-service-ca\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.794602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.794584 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-metrics-certs\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:09.855535 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:09.855433 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7j7\" (UniqueName: \"kubernetes.io/projected/690681d6-aa41-4b5f-9550-c9e41f620f9a-kube-api-access-wd7j7\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:10.296356 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:10.296316 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:10.299062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:10.299031 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls\") pod \"maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:10.527188 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:10.527155 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:10.911631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:10.911602 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4"] Nov 03 03:15:10.913800 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:15:10.913769 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690681d6_aa41_4b5f_9550_c9e41f620f9a.slice/crio-cff28dafda4253fe0fb14e30b37f706c14da5d6171ab0bde88b6a002775cd282 WatchSource:0}: Error finding container cff28dafda4253fe0fb14e30b37f706c14da5d6171ab0bde88b6a002775cd282: Status 404 returned error can't find the container with id cff28dafda4253fe0fb14e30b37f706c14da5d6171ab0bde88b6a002775cd282 Nov 03 03:15:11.317167 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:11.317130 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" event={"ID":"690681d6-aa41-4b5f-9550-c9e41f620f9a","Type":"ContainerStarted","Data":"cff28dafda4253fe0fb14e30b37f706c14da5d6171ab0bde88b6a002775cd282"} Nov 03 03:15:12.324022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:12.323982 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" event={"ID":"690681d6-aa41-4b5f-9550-c9e41f620f9a","Type":"ContainerStarted","Data":"3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12"} Nov 03 03:15:12.324414 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:12.324137 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:12.443914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:12.443860 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" podStartSLOduration=2.462514534 podStartE2EDuration="3.443844596s" totalImagesPullingTime="981.330062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:12.443595071 +0000 UTC m=+847.474193610" watchObservedRunningTime="2026-11-03 03:15:12.443844596 +0000 UTC m=+847.474443133" Nov 03 03:15:15.021923 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.021889 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts"] Nov 03 03:15:15.025603 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.025580 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.079177 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.079141 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4fb905b5-serving-cert\"" Nov 03 03:15:15.132248 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.132191 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts"] Nov 03 03:15:15.139740 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.139712 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.139891 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.139788 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd434bff-4e9a-4c03-8a45-57411d632e83-openshift-service-ca\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.139960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.139880 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgn2d\" (UniqueName: \"kubernetes.io/projected/bd434bff-4e9a-4c03-8a45-57411d632e83-kube-api-access-rgn2d\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.139960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.139925 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-metrics-certs\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.241098 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.241057 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd434bff-4e9a-4c03-8a45-57411d632e83-openshift-service-ca\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.241315 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.241115 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn2d\" (UniqueName: \"kubernetes.io/projected/bd434bff-4e9a-4c03-8a45-57411d632e83-kube-api-access-rgn2d\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.241315 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.241163 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-metrics-certs\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.241315 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.241209 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.241490 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:15.241406 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-serving-cert: secret "maas-api-e2e-migrate-4fb905b5-serving-cert" not found Nov 03 03:15:15.241576 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:15.241512 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls podName:bd434bff-4e9a-4c03-8a45-57411d632e83 nodeName:}" failed. No retries permitted until 2026-11-03 03:15:15.741490599 +0000 UTC m=+850.772089129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls") pod "maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" (UID: "bd434bff-4e9a-4c03-8a45-57411d632e83") : secret "maas-api-e2e-migrate-4fb905b5-serving-cert" not found Nov 03 03:15:15.242090 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.242051 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd434bff-4e9a-4c03-8a45-57411d632e83-openshift-service-ca\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.244519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.244481 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-metrics-certs\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.306119 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.306036 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn2d\" (UniqueName: \"kubernetes.io/projected/bd434bff-4e9a-4c03-8a45-57411d632e83-kube-api-access-rgn2d\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.745477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.745410 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.747999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.747975 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls\") pod \"maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:15.938793 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:15.938755 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:16.317225 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:16.317197 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts"] Nov 03 03:15:16.318878 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:15:16.318838 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd434bff_4e9a_4c03_8a45_57411d632e83.slice/crio-6596ad50d09663bc667ccf0d1991802d428d4b2dc4a440859098c707b9f89a83 WatchSource:0}: Error finding container 6596ad50d09663bc667ccf0d1991802d428d4b2dc4a440859098c707b9f89a83: Status 404 returned error can't find the container with id 6596ad50d09663bc667ccf0d1991802d428d4b2dc4a440859098c707b9f89a83 Nov 03 03:15:16.348554 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:16.348513 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" event={"ID":"bd434bff-4e9a-4c03-8a45-57411d632e83","Type":"ContainerStarted","Data":"6596ad50d09663bc667ccf0d1991802d428d4b2dc4a440859098c707b9f89a83"} Nov 03 03:15:17.357195 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:17.357159 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" event={"ID":"bd434bff-4e9a-4c03-8a45-57411d632e83","Type":"ContainerStarted","Data":"153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d"} Nov 03 03:15:17.357625 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:17.357296 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:17.477985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:17.477891 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" podStartSLOduration=2.674074155 podStartE2EDuration="3.477878185s" totalImagesPullingTime="803.80403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:17.477051072 +0000 UTC m=+852.507649609" watchObservedRunningTime="2026-11-03 03:15:17.477878185 +0000 UTC m=+852.508476722" Nov 03 03:15:18.334815 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:18.334785 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:15:23.369523 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:23.369496 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:15:30.012999 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:30.012957 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:15:30.190213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.190171 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf"] Nov 03 03:15:30.193788 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.193767 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.247169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.247135 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e21b4958-serving-cert\"" Nov 03 03:15:30.275762 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.275675 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5spt\" (UniqueName: \"kubernetes.io/projected/ffa166c1-2e6e-4879-9085-dd0ff8d97906-kube-api-access-x5spt\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.275929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.275832 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-metrics-certs\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.275929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.275870 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.275929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.275890 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa166c1-2e6e-4879-9085-dd0ff8d97906-openshift-service-ca\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.299914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.299876 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf"] Nov 03 03:15:30.376554 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.376519 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5spt\" (UniqueName: \"kubernetes.io/projected/ffa166c1-2e6e-4879-9085-dd0ff8d97906-kube-api-access-x5spt\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.376737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.376608 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-metrics-certs\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.376737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.376631 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.376737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.376647 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa166c1-2e6e-4879-9085-dd0ff8d97906-openshift-service-ca\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.376921 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:30.376757 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-serving-cert: secret "maas-api-e2e-ait-e21b4958-serving-cert" not found Nov 03 03:15:30.376921 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:15:30.376818 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls podName:ffa166c1-2e6e-4879-9085-dd0ff8d97906 nodeName:}" failed. No retries permitted until 2026-11-03 03:15:30.876802275 +0000 UTC m=+865.907400792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls") pod "maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" (UID: "ffa166c1-2e6e-4879-9085-dd0ff8d97906") : secret "maas-api-e2e-ait-e21b4958-serving-cert" not found Nov 03 03:15:30.377286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.377263 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa166c1-2e6e-4879-9085-dd0ff8d97906-openshift-service-ca\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.379268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.379240 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-metrics-certs\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.439561 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.439532 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5spt\" (UniqueName: \"kubernetes.io/projected/ffa166c1-2e6e-4879-9085-dd0ff8d97906-kube-api-access-x5spt\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.882401 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.882356 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:30.885035 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:30.885013 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls\") pod \"maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:31.107432 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:31.107395 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:31.487355 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:31.487323 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf"] Nov 03 03:15:31.488857 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:15:31.488820 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa166c1_2e6e_4879_9085_dd0ff8d97906.slice/crio-1e46c9c0aa09ea6005331683300a3101ca86480ce055d83953dea178576435a3 WatchSource:0}: Error finding container 1e46c9c0aa09ea6005331683300a3101ca86480ce055d83953dea178576435a3: Status 404 returned error can't find the container with id 1e46c9c0aa09ea6005331683300a3101ca86480ce055d83953dea178576435a3 Nov 03 03:15:32.437046 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:32.437010 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" event={"ID":"ffa166c1-2e6e-4879-9085-dd0ff8d97906","Type":"ContainerStarted","Data":"bdffe624b5f3b4436eceae383eff87071bd57402b46b2451ae3055b6957bcd9a"} Nov 03 03:15:32.437433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:32.437079 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" event={"ID":"ffa166c1-2e6e-4879-9085-dd0ff8d97906","Type":"ContainerStarted","Data":"1e46c9c0aa09ea6005331683300a3101ca86480ce055d83953dea178576435a3"} Nov 03 03:15:32.437433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:32.437106 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:15:32.561256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:32.561198 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" podStartSLOduration=1.732706004 podStartE2EDuration="2.561184997s" totalImagesPullingTime="828.478993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:32.559745649 +0000 UTC m=+867.590344187" watchObservedRunningTime="2026-11-03 03:15:32.561184997 +0000 UTC m=+867.591783550" Nov 03 03:15:38.447361 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:15:38.447332 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:16:00.011480 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:00.011414 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:16:05.181403 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:05.181372 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:16:05.185560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:05.185538 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:16:10.993385 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:10.993348 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4"] Nov 03 03:16:10.993989 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:10.993584 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" podUID="690681d6-aa41-4b5f-9550-c9e41f620f9a" containerName="maas-api" containerID="cri-o://3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12" gracePeriod=30 Nov 03 03:16:11.255006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.254942 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:16:11.364257 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.364221 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690681d6-aa41-4b5f-9550-c9e41f620f9a-kube-api-access-wd7j7\" (UniqueName: \"kubernetes.io/projected/690681d6-aa41-4b5f-9550-c9e41f620f9a-kube-api-access-wd7j7\") pod \"690681d6-aa41-4b5f-9550-c9e41f620f9a\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " Nov 03 03:16:11.364485 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.364327 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-metrics-certs\") pod \"690681d6-aa41-4b5f-9550-c9e41f620f9a\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " Nov 03 03:16:11.364485 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.364364 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690681d6-aa41-4b5f-9550-c9e41f620f9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690681d6-aa41-4b5f-9550-c9e41f620f9a-openshift-service-ca\") pod \"690681d6-aa41-4b5f-9550-c9e41f620f9a\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " Nov 03 03:16:11.364485 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.364397 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls\") pod \"690681d6-aa41-4b5f-9550-c9e41f620f9a\" (UID: \"690681d6-aa41-4b5f-9550-c9e41f620f9a\") " Nov 03 03:16:11.364759 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.364730 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690681d6-aa41-4b5f-9550-c9e41f620f9a-openshift-service-ca" pod "690681d6-aa41-4b5f-9550-c9e41f620f9a" (UID: "690681d6-aa41-4b5f-9550-c9e41f620f9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:11.366972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.366939 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls" pod "690681d6-aa41-4b5f-9550-c9e41f620f9a" (UID: "690681d6-aa41-4b5f-9550-c9e41f620f9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:11.367205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.367185 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690681d6-aa41-4b5f-9550-c9e41f620f9a-kube-api-access-wd7j7" pod "690681d6-aa41-4b5f-9550-c9e41f620f9a" (UID: "690681d6-aa41-4b5f-9550-c9e41f620f9a"). InnerVolumeSpecName "kube-api-access-wd7j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:11.367205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.367187 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-metrics-certs" pod "690681d6-aa41-4b5f-9550-c9e41f620f9a" (UID: "690681d6-aa41-4b5f-9550-c9e41f620f9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:11.465655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.465619 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:11.465655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.465647 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/690681d6-aa41-4b5f-9550-c9e41f620f9a-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:11.465655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.465657 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/690681d6-aa41-4b5f-9550-c9e41f620f9a-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:11.465655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.465666 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7j7\" (UniqueName: \"kubernetes.io/projected/690681d6-aa41-4b5f-9550-c9e41f620f9a-kube-api-access-wd7j7\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:11.644780 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.644739 5117 generic.go:356] "Generic (PLEG): container finished" podID="690681d6-aa41-4b5f-9550-c9e41f620f9a" containerID="3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12" exitCode=0 Nov 03 03:16:11.644968 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.644825 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" Nov 03 03:16:11.644968 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.644835 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" event={"ID":"690681d6-aa41-4b5f-9550-c9e41f620f9a","Type":"ContainerDied","Data":"3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12"} Nov 03 03:16:11.644968 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.644920 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4" event={"ID":"690681d6-aa41-4b5f-9550-c9e41f620f9a","Type":"ContainerDied","Data":"cff28dafda4253fe0fb14e30b37f706c14da5d6171ab0bde88b6a002775cd282"} Nov 03 03:16:11.644968 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.644943 5117 scope.go:122] "RemoveContainer" containerID="3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12" Nov 03 03:16:11.658023 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.658006 5117 scope.go:122] "RemoveContainer" containerID="3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12" Nov 03 03:16:11.658382 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:11.658357 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12\": container with ID starting with 3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12 not found: ID does not exist" containerID="3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12" Nov 03 03:16:11.658433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.658398 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12"} err="failed to get container status \"3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12\": rpc error: code = NotFound desc = could not find container \"3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12\": container with ID starting with 3716807ad8f54c88de416562eefb4da1d6a23931c55c447c0e1425d220f46d12 not found: ID does not exist" Nov 03 03:16:11.811978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.811942 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4"] Nov 03 03:16:11.862973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:11.862935 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b4ff4779-58cd675b97-mh8r4"] Nov 03 03:16:12.243635 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.243597 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts"] Nov 03 03:16:12.244029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.243807 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" podUID="bd434bff-4e9a-4c03-8a45-57411d632e83" containerName="maas-api" containerID="cri-o://153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d" gracePeriod=30 Nov 03 03:16:12.505277 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.505206 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:16:12.655015 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.654974 5117 generic.go:356] "Generic (PLEG): container finished" podID="bd434bff-4e9a-4c03-8a45-57411d632e83" containerID="153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d" exitCode=0 Nov 03 03:16:12.655208 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.655038 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" Nov 03 03:16:12.655208 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.655039 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" event={"ID":"bd434bff-4e9a-4c03-8a45-57411d632e83","Type":"ContainerDied","Data":"153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d"} Nov 03 03:16:12.655208 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.655172 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts" event={"ID":"bd434bff-4e9a-4c03-8a45-57411d632e83","Type":"ContainerDied","Data":"6596ad50d09663bc667ccf0d1991802d428d4b2dc4a440859098c707b9f89a83"} Nov 03 03:16:12.655208 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.655209 5117 scope.go:122] "RemoveContainer" containerID="153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d" Nov 03 03:16:12.665880 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.665857 5117 scope.go:122] "RemoveContainer" containerID="153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d" Nov 03 03:16:12.666233 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:12.666209 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d\": container with ID starting with 153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d not found: ID does not exist" containerID="153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d" Nov 03 03:16:12.666281 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.666244 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d"} err="failed to get container status \"153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d\": rpc error: code = NotFound desc = could not find container \"153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d\": container with ID starting with 153686b21de2c73887b47a906ab0567fcad626ade84f2b601f524abdf0b3fc0d not found: ID does not exist" Nov 03 03:16:12.678608 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.678576 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd434bff-4e9a-4c03-8a45-57411d632e83-kube-api-access-rgn2d\" (UniqueName: \"kubernetes.io/projected/bd434bff-4e9a-4c03-8a45-57411d632e83-kube-api-access-rgn2d\") pod \"bd434bff-4e9a-4c03-8a45-57411d632e83\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " Nov 03 03:16:12.678717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.678636 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-metrics-certs\") pod \"bd434bff-4e9a-4c03-8a45-57411d632e83\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " Nov 03 03:16:12.678753 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.678738 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd434bff-4e9a-4c03-8a45-57411d632e83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd434bff-4e9a-4c03-8a45-57411d632e83-openshift-service-ca\") pod \"bd434bff-4e9a-4c03-8a45-57411d632e83\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " Nov 03 03:16:12.678786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.678759 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls\") pod \"bd434bff-4e9a-4c03-8a45-57411d632e83\" (UID: \"bd434bff-4e9a-4c03-8a45-57411d632e83\") " Nov 03 03:16:12.679103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.679074 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd434bff-4e9a-4c03-8a45-57411d632e83-openshift-service-ca" pod "bd434bff-4e9a-4c03-8a45-57411d632e83" (UID: "bd434bff-4e9a-4c03-8a45-57411d632e83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:12.681019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.680993 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd434bff-4e9a-4c03-8a45-57411d632e83-kube-api-access-rgn2d" pod "bd434bff-4e9a-4c03-8a45-57411d632e83" (UID: "bd434bff-4e9a-4c03-8a45-57411d632e83"). InnerVolumeSpecName "kube-api-access-rgn2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:12.681112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.681067 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-metrics-certs" pod "bd434bff-4e9a-4c03-8a45-57411d632e83" (UID: "bd434bff-4e9a-4c03-8a45-57411d632e83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:12.681112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.681086 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls" pod "bd434bff-4e9a-4c03-8a45-57411d632e83" (UID: "bd434bff-4e9a-4c03-8a45-57411d632e83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:12.779935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.779842 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgn2d\" (UniqueName: \"kubernetes.io/projected/bd434bff-4e9a-4c03-8a45-57411d632e83-kube-api-access-rgn2d\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:12.779935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.779875 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:12.779935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.779890 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd434bff-4e9a-4c03-8a45-57411d632e83-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:12.779935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:12.779902 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd434bff-4e9a-4c03-8a45-57411d632e83-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:13.079321 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:13.079269 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts"] Nov 03 03:16:13.088909 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:13.088879 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690681d6-aa41-4b5f-9550-c9e41f620f9a" path="/var/lib/kubelet/pods/690681d6-aa41-4b5f-9550-c9e41f620f9a/volumes" Nov 03 03:16:13.130031 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:13.129993 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4fb905b5-86758b5fb6-2vnts"] Nov 03 03:16:15.090481 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:15.090422 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd434bff-4e9a-4c03-8a45-57411d632e83" path="/var/lib/kubelet/pods/bd434bff-4e9a-4c03-8a45-57411d632e83/volumes" Nov 03 03:16:24.563388 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.563352 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf"] Nov 03 03:16:24.563937 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.563571 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" podUID="ffa166c1-2e6e-4879-9085-dd0ff8d97906" containerName="maas-api" containerID="cri-o://bdffe624b5f3b4436eceae383eff87071bd57402b46b2451ae3055b6957bcd9a" gracePeriod=30 Nov 03 03:16:24.725509 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.725424 5117 generic.go:356] "Generic (PLEG): container finished" podID="ffa166c1-2e6e-4879-9085-dd0ff8d97906" containerID="bdffe624b5f3b4436eceae383eff87071bd57402b46b2451ae3055b6957bcd9a" exitCode=0 Nov 03 03:16:24.725699 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.725507 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" event={"ID":"ffa166c1-2e6e-4879-9085-dd0ff8d97906","Type":"ContainerDied","Data":"bdffe624b5f3b4436eceae383eff87071bd57402b46b2451ae3055b6957bcd9a"} Nov 03 03:16:24.831364 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.831338 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:16:24.986986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.986951 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-metrics-certs\") pod \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " Nov 03 03:16:24.987172 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.987006 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa166c1-2e6e-4879-9085-dd0ff8d97906-kube-api-access-x5spt\" (UniqueName: \"kubernetes.io/projected/ffa166c1-2e6e-4879-9085-dd0ff8d97906-kube-api-access-x5spt\") pod \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " Nov 03 03:16:24.987172 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.987032 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls\") pod \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " Nov 03 03:16:24.987261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.987222 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa166c1-2e6e-4879-9085-dd0ff8d97906-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa166c1-2e6e-4879-9085-dd0ff8d97906-openshift-service-ca\") pod \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\" (UID: \"ffa166c1-2e6e-4879-9085-dd0ff8d97906\") " Nov 03 03:16:24.987646 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.987608 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa166c1-2e6e-4879-9085-dd0ff8d97906-openshift-service-ca" pod "ffa166c1-2e6e-4879-9085-dd0ff8d97906" (UID: "ffa166c1-2e6e-4879-9085-dd0ff8d97906"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:16:24.989376 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.989341 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls" pod "ffa166c1-2e6e-4879-9085-dd0ff8d97906" (UID: "ffa166c1-2e6e-4879-9085-dd0ff8d97906"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:24.989376 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.989369 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-metrics-certs" pod "ffa166c1-2e6e-4879-9085-dd0ff8d97906" (UID: "ffa166c1-2e6e-4879-9085-dd0ff8d97906"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:16:24.989869 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:24.989402 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa166c1-2e6e-4879-9085-dd0ff8d97906-kube-api-access-x5spt" pod "ffa166c1-2e6e-4879-9085-dd0ff8d97906" (UID: "ffa166c1-2e6e-4879-9085-dd0ff8d97906"). InnerVolumeSpecName "kube-api-access-x5spt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:16:25.088001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.087919 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5spt\" (UniqueName: \"kubernetes.io/projected/ffa166c1-2e6e-4879-9085-dd0ff8d97906-kube-api-access-x5spt\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:25.088001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.087945 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:25.088001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.087956 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa166c1-2e6e-4879-9085-dd0ff8d97906-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:25.088001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.087964 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa166c1-2e6e-4879-9085-dd0ff8d97906-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:16:25.731851 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.731817 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" Nov 03 03:16:25.731851 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.731835 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf" event={"ID":"ffa166c1-2e6e-4879-9085-dd0ff8d97906","Type":"ContainerDied","Data":"1e46c9c0aa09ea6005331683300a3101ca86480ce055d83953dea178576435a3"} Nov 03 03:16:25.732469 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.731938 5117 scope.go:122] "RemoveContainer" containerID="bdffe624b5f3b4436eceae383eff87071bd57402b46b2451ae3055b6957bcd9a" Nov 03 03:16:25.899588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.899547 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf"] Nov 03 03:16:25.950199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:25.950163 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e21b4958-7fcf89d69b-4rsbf"] Nov 03 03:16:27.088493 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:27.088462 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa166c1-2e6e-4879-9085-dd0ff8d97906" path="/var/lib/kubelet/pods/ffa166c1-2e6e-4879-9085-dd0ff8d97906/volumes" Nov 03 03:16:30.011133 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:30.011099 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:16:32.329980 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.329945 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s"] Nov 03 03:16:32.330369 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330326 5117 cpu_manager.go:414] "containerMap: removing container" podUID="bd434bff-4e9a-4c03-8a45-57411d632e83" containerName="maas-api" Nov 03 03:16:32.330369 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330337 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd434bff-4e9a-4c03-8a45-57411d632e83" containerName="maas-api" Nov 03 03:16:32.330369 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330350 5117 cpu_manager.go:414] "containerMap: removing container" podUID="ffa166c1-2e6e-4879-9085-dd0ff8d97906" containerName="maas-api" Nov 03 03:16:32.330369 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330355 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa166c1-2e6e-4879-9085-dd0ff8d97906" containerName="maas-api" Nov 03 03:16:32.330369 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330365 5117 cpu_manager.go:414] "containerMap: removing container" podUID="690681d6-aa41-4b5f-9550-c9e41f620f9a" containerName="maas-api" Nov 03 03:16:32.330369 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330370 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690681d6-aa41-4b5f-9550-c9e41f620f9a" containerName="maas-api" Nov 03 03:16:32.330602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330478 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd434bff-4e9a-4c03-8a45-57411d632e83" containerName="maas-api" Nov 03 03:16:32.330602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330486 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa166c1-2e6e-4879-9085-dd0ff8d97906" containerName="maas-api" Nov 03 03:16:32.330602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.330493 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="690681d6-aa41-4b5f-9550-c9e41f620f9a" containerName="maas-api" Nov 03 03:16:32.334836 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.334818 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.348955 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.348928 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-metrics-certs\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.349101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.348965 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c995c-b622-474e-ad2b-1c70fa3a6d86-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.349101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.348990 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hd2\" (UniqueName: \"kubernetes.io/projected/569c995c-b622-474e-ad2b-1c70fa3a6d86-kube-api-access-79hd2\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.349217 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.349133 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.441658 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.441624 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-de27f5-serving-cert\"" Nov 03 03:16:32.441838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.441694 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 03:16:32.441838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.441707 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcxk\"" Nov 03 03:16:32.444989 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.444963 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s"] Nov 03 03:16:32.450264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.450240 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-metrics-certs\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.450364 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.450287 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c995c-b622-474e-ad2b-1c70fa3a6d86-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.450557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.450477 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hd2\" (UniqueName: \"kubernetes.io/projected/569c995c-b622-474e-ad2b-1c70fa3a6d86-kube-api-access-79hd2\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.450557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.450540 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.450710 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:32.450684 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-serving-cert: secret "maas-api-e2e-worker-w2-de27f5-serving-cert" not found Nov 03 03:16:32.450825 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:32.450771 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls podName:569c995c-b622-474e-ad2b-1c70fa3a6d86 nodeName:}" failed. No retries permitted until 2026-11-03 03:16:32.950751561 +0000 UTC m=+927.981350078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls") pod "maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" (UID: "569c995c-b622-474e-ad2b-1c70fa3a6d86") : secret "maas-api-e2e-worker-w2-de27f5-serving-cert" not found Nov 03 03:16:32.451103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.451081 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c995c-b622-474e-ad2b-1c70fa3a6d86-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.453346 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.453314 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-metrics-certs\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.514536 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.514500 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hd2\" (UniqueName: \"kubernetes.io/projected/569c995c-b622-474e-ad2b-1c70fa3a6d86-kube-api-access-79hd2\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.953767 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.953734 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:32.956381 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:32.956359 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls\") pod \"maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:33.247616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:33.247537 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:33.623854 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:33.623829 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s"] Nov 03 03:16:33.625669 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:16:33.625633 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c995c_b622_474e_ad2b_1c70fa3a6d86.slice/crio-c170bc0129a8f0599f037119812b4925ae53907f7d1a3b2c69d1fd48c628faa1 WatchSource:0}: Error finding container c170bc0129a8f0599f037119812b4925ae53907f7d1a3b2c69d1fd48c628faa1: Status 404 returned error can't find the container with id c170bc0129a8f0599f037119812b4925ae53907f7d1a3b2c69d1fd48c628faa1 Nov 03 03:16:33.777433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:33.777399 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" event={"ID":"569c995c-b622-474e-ad2b-1c70fa3a6d86","Type":"ContainerStarted","Data":"c170bc0129a8f0599f037119812b4925ae53907f7d1a3b2c69d1fd48c628faa1"} Nov 03 03:16:34.788898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:34.788860 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" event={"ID":"569c995c-b622-474e-ad2b-1c70fa3a6d86","Type":"ContainerStarted","Data":"592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf"} Nov 03 03:16:34.789384 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:34.788983 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:34.911985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:34.911879 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" podStartSLOduration=1.9521087160000001 podStartE2EDuration="2.911865549s" totalImagesPullingTime="959.756833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:34.911032711 +0000 UTC m=+929.941631249" watchObservedRunningTime="2026-11-03 03:16:34.911865549 +0000 UTC m=+929.942464087" Nov 03 03:16:37.888159 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:37.888075 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk"] Nov 03 03:16:37.891887 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:37.891869 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:37.945704 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:37.945667 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-68b670-serving-cert\"" Nov 03 03:16:37.952027 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:37.951999 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk"] Nov 03 03:16:38.000022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:37.999987 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7mz8\" (UniqueName: \"kubernetes.io/projected/392157ef-9f8e-48b6-bdcd-da14bce0f10d-kube-api-access-x7mz8\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.000218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.000023 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-metrics-certs\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.000218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.000127 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.000324 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.000216 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/392157ef-9f8e-48b6-bdcd-da14bce0f10d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.101542 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.101437 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7mz8\" (UniqueName: \"kubernetes.io/projected/392157ef-9f8e-48b6-bdcd-da14bce0f10d-kube-api-access-x7mz8\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.101735 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.101579 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-metrics-certs\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.101822 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.101760 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.101822 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.101802 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/392157ef-9f8e-48b6-bdcd-da14bce0f10d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.102096 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:38.102072 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-serving-cert: secret "maas-api-e2e-worker-w3-68b670-serving-cert" not found Nov 03 03:16:38.102203 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:16:38.102158 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls podName:392157ef-9f8e-48b6-bdcd-da14bce0f10d nodeName:}" failed. No retries permitted until 2026-11-03 03:16:38.602134803 +0000 UTC m=+933.632733325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls") pod "maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" (UID: "392157ef-9f8e-48b6-bdcd-da14bce0f10d") : secret "maas-api-e2e-worker-w3-68b670-serving-cert" not found Nov 03 03:16:38.102549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.102526 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/392157ef-9f8e-48b6-bdcd-da14bce0f10d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.104581 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.104559 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-metrics-certs\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.164420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.164349 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7mz8\" (UniqueName: \"kubernetes.io/projected/392157ef-9f8e-48b6-bdcd-da14bce0f10d-kube-api-access-x7mz8\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.605785 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.605743 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.608456 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.608410 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:38.804588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:38.804549 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:39.181661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:39.181634 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk"] Nov 03 03:16:39.183179 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:16:39.183151 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392157ef_9f8e_48b6_bdcd_da14bce0f10d.slice/crio-38ce44be34e678e6288d687ffb2cc23b3b534d096b75000e43b16fec9c88a440 WatchSource:0}: Error finding container 38ce44be34e678e6288d687ffb2cc23b3b534d096b75000e43b16fec9c88a440: Status 404 returned error can't find the container with id 38ce44be34e678e6288d687ffb2cc23b3b534d096b75000e43b16fec9c88a440 Nov 03 03:16:39.814651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:39.814617 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" event={"ID":"392157ef-9f8e-48b6-bdcd-da14bce0f10d","Type":"ContainerStarted","Data":"38ce44be34e678e6288d687ffb2cc23b3b534d096b75000e43b16fec9c88a440"} Nov 03 03:16:40.799165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:40.799138 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 03:16:40.821328 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:40.821294 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" event={"ID":"392157ef-9f8e-48b6-bdcd-da14bce0f10d","Type":"ContainerStarted","Data":"c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8"} Nov 03 03:16:40.821481 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:40.821354 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:16:41.052870 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:41.052770 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" podStartSLOduration=3.221341931 podStartE2EDuration="4.052754831s" totalImagesPullingTime="831.4129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:41.050894514 +0000 UTC m=+936.081493052" watchObservedRunningTime="2026-11-03 03:16:41.052754831 +0000 UTC m=+936.083353369" Nov 03 03:16:46.832069 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:16:46.832041 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 03:17:00.011723 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:17:00.011655 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:17:17.772962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.772923 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g"] Nov 03 03:17:17.776613 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.776596 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.830165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.830125 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5a6246-serving-cert\"" Nov 03 03:17:17.871661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.871623 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a3d918-5836-4a33-b411-a05af4647b7b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.871848 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.871678 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.871848 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.871752 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kr8q\" (UniqueName: \"kubernetes.io/projected/73a3d918-5836-4a33-b411-a05af4647b7b-kube-api-access-9kr8q\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.871980 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.871893 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-metrics-certs\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.882999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.882964 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g"] Nov 03 03:17:17.972911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.972882 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-metrics-certs\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.973070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.972950 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a3d918-5836-4a33-b411-a05af4647b7b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.973070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.972975 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.973070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.972993 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kr8q\" (UniqueName: \"kubernetes.io/projected/73a3d918-5836-4a33-b411-a05af4647b7b-kube-api-access-9kr8q\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.973188 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:17:17.973113 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-serving-cert: secret "maas-api-e2e-worker-w1-5a6246-serving-cert" not found Nov 03 03:17:17.973188 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:17:17.973176 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls podName:73a3d918-5836-4a33-b411-a05af4647b7b nodeName:}" failed. No retries permitted until 2026-11-03 03:17:18.473156736 +0000 UTC m=+973.503755255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls") pod "maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" (UID: "73a3d918-5836-4a33-b411-a05af4647b7b") : secret "maas-api-e2e-worker-w1-5a6246-serving-cert" not found Nov 03 03:17:17.973682 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.973661 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a3d918-5836-4a33-b411-a05af4647b7b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:17.975479 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:17.975458 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-metrics-certs\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:18.037833 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:18.037750 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kr8q\" (UniqueName: \"kubernetes.io/projected/73a3d918-5836-4a33-b411-a05af4647b7b-kube-api-access-9kr8q\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:18.477557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:18.477512 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:18.480345 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:18.480317 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:18.689346 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:18.689305 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:19.068331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:19.068302 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g"] Nov 03 03:17:19.070405 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:17:19.070375 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a3d918_5836_4a33_b411_a05af4647b7b.slice/crio-3f3ace1939f7653b3cb20413ac78efb742e508eb6fbb5146511f1142ebc64b76 WatchSource:0}: Error finding container 3f3ace1939f7653b3cb20413ac78efb742e508eb6fbb5146511f1142ebc64b76: Status 404 returned error can't find the container with id 3f3ace1939f7653b3cb20413ac78efb742e508eb6fbb5146511f1142ebc64b76 Nov 03 03:17:20.027104 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:20.027055 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" event={"ID":"73a3d918-5836-4a33-b411-a05af4647b7b","Type":"ContainerStarted","Data":"cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee"} Nov 03 03:17:20.027277 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:20.027128 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" event={"ID":"73a3d918-5836-4a33-b411-a05af4647b7b","Type":"ContainerStarted","Data":"3f3ace1939f7653b3cb20413ac78efb742e508eb6fbb5146511f1142ebc64b76"} Nov 03 03:17:20.027326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:20.027275 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:20.199317 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:20.199206 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" podStartSLOduration=2.370398861 podStartE2EDuration="3.199189077s" totalImagesPullingTime="828.790216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:20.197174218 +0000 UTC m=+975.227772758" watchObservedRunningTime="2026-11-03 03:17:20.199189077 +0000 UTC m=+975.229787614" Nov 03 03:17:22.879661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.879626 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc"] Nov 03 03:17:22.883963 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.883942 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:22.918411 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.918381 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvz7x\" (UniqueName: \"kubernetes.io/projected/6b43ac6c-1194-4622-b271-a669efeb6588-kube-api-access-dvz7x\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:22.918562 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.918430 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:22.918624 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.918601 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43ac6c-1194-4622-b271-a669efeb6588-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:22.918661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.918639 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-metrics-certs\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:22.991100 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.991072 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b1ad4e-serving-cert\"" Nov 03 03:17:22.993504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:22.993482 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc"] Nov 03 03:17:23.019604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.019574 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43ac6c-1194-4622-b271-a669efeb6588-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.019754 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.019620 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-metrics-certs\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.019754 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.019665 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvz7x\" (UniqueName: \"kubernetes.io/projected/6b43ac6c-1194-4622-b271-a669efeb6588-kube-api-access-dvz7x\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.019754 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.019727 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.019926 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:17:23.019874 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-serving-cert: secret "maas-api-e2e-worker-w4-b1ad4e-serving-cert" not found Nov 03 03:17:23.019983 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:17:23.019945 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls podName:6b43ac6c-1194-4622-b271-a669efeb6588 nodeName:}" failed. No retries permitted until 2026-11-03 03:17:23.519923255 +0000 UTC m=+978.550521793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls") pod "maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" (UID: "6b43ac6c-1194-4622-b271-a669efeb6588") : secret "maas-api-e2e-worker-w4-b1ad4e-serving-cert" not found Nov 03 03:17:23.020305 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.020283 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43ac6c-1194-4622-b271-a669efeb6588-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.022721 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.022698 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-metrics-certs\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.082788 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.082755 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvz7x\" (UniqueName: \"kubernetes.io/projected/6b43ac6c-1194-4622-b271-a669efeb6588-kube-api-access-dvz7x\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.524722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.524692 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.527234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.527208 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:23.797397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:23.797314 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:24.176642 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:24.176618 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc"] Nov 03 03:17:24.178060 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:17:24.178032 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b43ac6c_1194_4622_b271_a669efeb6588.slice/crio-599c25b590998535e762c558b8f5612c8eef38fbc07978ac4ff7313fdc611c1f WatchSource:0}: Error finding container 599c25b590998535e762c558b8f5612c8eef38fbc07978ac4ff7313fdc611c1f: Status 404 returned error can't find the container with id 599c25b590998535e762c558b8f5612c8eef38fbc07978ac4ff7313fdc611c1f Nov 03 03:17:25.057339 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:25.057297 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" event={"ID":"6b43ac6c-1194-4622-b271-a669efeb6588","Type":"ContainerStarted","Data":"7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e"} Nov 03 03:17:25.057553 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:25.057354 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" event={"ID":"6b43ac6c-1194-4622-b271-a669efeb6588","Type":"ContainerStarted","Data":"599c25b590998535e762c558b8f5612c8eef38fbc07978ac4ff7313fdc611c1f"} Nov 03 03:17:25.057553 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:25.057471 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:17:25.228089 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:25.228034 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" podStartSLOduration=2.424367643 podStartE2EDuration="3.228020391s" totalImagesPullingTime="803.652748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:25.227183662 +0000 UTC m=+980.257782199" watchObservedRunningTime="2026-11-03 03:17:25.228020391 +0000 UTC m=+980.258618928" Nov 03 03:17:26.038105 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:26.038075 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 03:17:30.010787 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:17:30.010752 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:17:31.067329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:17:31.067297 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 03:18:00.011079 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:18:00.011044 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:18:30.010312 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:18:30.010222 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:19:00.010593 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:19:00.010491 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:19:24.259305 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.259271 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs"] Nov 03 03:19:24.262968 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.262950 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.320649 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.320617 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-home\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.320649 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.320660 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-tmp-dir\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.320889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.320754 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-dshm\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.320889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.320849 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8333631b-54aa-4041-a8c6-a725077a336f-tls-certs\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.320978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.320926 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-model-cache\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.320978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.320943 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5jg\" (UniqueName: \"kubernetes.io/projected/8333631b-54aa-4041-a8c6-a725077a336f-kube-api-access-cq5jg\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.369906 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.369875 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"kube-root-ca.crt\"" Nov 03 03:19:24.370075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.370008 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"openshift-service-ca.crt\"" Nov 03 03:19:24.370075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.370014 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"default-dockercfg-rjr5x\"" Nov 03 03:19:24.370075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.370060 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"premium-w1-5a6246-kserve-self-signed-certs\"" Nov 03 03:19:24.372492 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.372472 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs"] Nov 03 03:19:24.422211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422179 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-home\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422215 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-tmp-dir\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422238 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-dshm\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422282 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8333631b-54aa-4041-a8c6-a725077a336f-tls-certs\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422360 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-model-cache\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422382 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5jg\" (UniqueName: \"kubernetes.io/projected/8333631b-54aa-4041-a8c6-a725077a336f-kube-api-access-cq5jg\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422669 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422637 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-home\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422674 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-tmp-dir\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.422826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.422805 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-model-cache\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.424726 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.424701 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-dshm\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.424960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.424946 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8333631b-54aa-4041-a8c6-a725077a336f-tls-certs\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.484968 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.484941 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5jg\" (UniqueName: \"kubernetes.io/projected/8333631b-54aa-4041-a8c6-a725077a336f-kube-api-access-cq5jg\") pod \"premium-w1-5a6246-kserve-d4564db9-9nffs\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.574234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.574204 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:24.968782 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.968749 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs"] Nov 03 03:19:24.970700 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:19:24.970663 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8333631b_54aa_4041_a8c6_a725077a336f.slice/crio-a2a11e6cc64af4038a73ceb8ab8bccd5688fa6699b2ba8eedcfd67522ff7db94 WatchSource:0}: Error finding container a2a11e6cc64af4038a73ceb8ab8bccd5688fa6699b2ba8eedcfd67522ff7db94: Status 404 returned error can't find the container with id a2a11e6cc64af4038a73ceb8ab8bccd5688fa6699b2ba8eedcfd67522ff7db94 Nov 03 03:19:24.972629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:24.972610 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:19:25.697980 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:25.697939 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" event={"ID":"8333631b-54aa-4041-a8c6-a725077a336f","Type":"ContainerStarted","Data":"a2a11e6cc64af4038a73ceb8ab8bccd5688fa6699b2ba8eedcfd67522ff7db94"} Nov 03 03:19:27.710436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:27.710396 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" event={"ID":"8333631b-54aa-4041-a8c6-a725077a336f","Type":"ContainerStarted","Data":"a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa"} Nov 03 03:19:27.830904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:27.830834 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podStartSLOduration=1.82674274 podStartE2EDuration="3.830815629s" totalImagesPullingTime="2.004072889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:19:27.830393513 +0000 UTC m=+1102.860992051" watchObservedRunningTime="2026-11-03 03:19:27.830815629 +0000 UTC m=+1102.861414168" Nov 03 03:19:28.574517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:28.574484 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:30.016283 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:19:30.013840 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:19:34.574967 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:34.574932 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:34.587214 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:34.587187 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:34.757095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:34.757066 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:19:53.177054 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.176974 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bwmn6-ltxbv"] Nov 03 03:19:53.180643 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.180627 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.200827 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.200802 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-trusted-ca-bundle\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.200929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.200862 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e7c24f96-ab39-4403-9251-27cceb0c2385-archives-path\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.200970 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.200952 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9fk\" (UniqueName: \"kubernetes.io/projected/e7c24f96-ab39-4403-9251-27cceb0c2385-kube-api-access-hd9fk\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.201004 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.200981 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-service-ca-bundle\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.236319 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.236288 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vbzfk\"" Nov 03 03:19:53.238697 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.238668 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bwmn6-ltxbv"] Nov 03 03:19:53.301525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.301494 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9fk\" (UniqueName: \"kubernetes.io/projected/e7c24f96-ab39-4403-9251-27cceb0c2385-kube-api-access-hd9fk\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.301683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.301537 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-service-ca-bundle\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.301683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.301579 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-trusted-ca-bundle\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.301683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.301668 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e7c24f96-ab39-4403-9251-27cceb0c2385-archives-path\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.302108 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.302086 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e7c24f96-ab39-4403-9251-27cceb0c2385-archives-path\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.302299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.302284 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-service-ca-bundle\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.302362 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.302342 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-trusted-ca-bundle\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.365590 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.365563 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9fk\" (UniqueName: \"kubernetes.io/projected/e7c24f96-ab39-4403-9251-27cceb0c2385-kube-api-access-hd9fk\") pod \"periodic-gathering-bwmn6-ltxbv\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.492785 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.492709 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:19:53.868308 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:53.868285 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bwmn6-ltxbv"] Nov 03 03:19:53.871108 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:19:53.871076 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c24f96_ab39_4403_9251_27cceb0c2385.slice/crio-a12680e31875ba38adc741a1e16426bb8b8cb002eab1c73ab01d5aa8038a9df6 WatchSource:0}: Error finding container a12680e31875ba38adc741a1e16426bb8b8cb002eab1c73ab01d5aa8038a9df6: Status 404 returned error can't find the container with id a12680e31875ba38adc741a1e16426bb8b8cb002eab1c73ab01d5aa8038a9df6 Nov 03 03:19:54.854593 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:54.854549 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" event={"ID":"e7c24f96-ab39-4403-9251-27cceb0c2385","Type":"ContainerStarted","Data":"1f8a5d8e8b9e6f4766dc8d09a2a91fa68799b072a53c89b98c98b9bc01c3feac"} Nov 03 03:19:54.855120 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:54.854629 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" event={"ID":"e7c24f96-ab39-4403-9251-27cceb0c2385","Type":"ContainerStarted","Data":"a12680e31875ba38adc741a1e16426bb8b8cb002eab1c73ab01d5aa8038a9df6"} Nov 03 03:19:54.973847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:19:54.973780 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" podStartSLOduration=1.973757905 podStartE2EDuration="1.973757905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:19:54.972038802 +0000 UTC m=+1130.002637344" watchObservedRunningTime="2026-11-03 03:19:54.973757905 +0000 UTC m=+1130.004356446" Nov 03 03:20:00.011257 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:20:00.011221 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:20:30.010328 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:20:30.010288 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:20:49.391161 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.391123 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn"] Nov 03 03:20:49.394737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.394720 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437061 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437020 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437088 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437220 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437275 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437322 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437359 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437528 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b65jf\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-kube-api-access-b65jf\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437568 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.437654 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.437637 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.492820 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.492766 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn"] Nov 03 03:20:49.538298 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538253 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b65jf\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-kube-api-access-b65jf\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538304 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538342 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538404 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538483 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538767 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538539 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538767 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538595 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538767 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538624 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.538767 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.538652 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.539390 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.539123 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.539827 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.539801 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.540826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.540798 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.541055 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.541033 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.541730 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.541680 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.543016 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.542991 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.543485 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.543427 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.603475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.603413 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b65jf\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-kube-api-access-b65jf\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.603660 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.603583 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-kjzbn\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.707966 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.707875 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:49.956642 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.956606 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn"] Nov 03 03:20:49.961377 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:49.961305 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.016481 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.016409 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"default-dockercfg-qtmq4\"" Nov 03 03:20:50.016678 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.016487 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"openshift-service-ca.crt\"" Nov 03 03:20:50.016678 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.016425 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"e2e-conflict-a-e5f308a1-kserve-self-signed-certs\"" Nov 03 03:20:50.016678 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.016548 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"kube-root-ca.crt\"" Nov 03 03:20:50.019001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.018974 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn"] Nov 03 03:20:50.042405 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.042369 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e90cb3-0324-4828-be59-44040eb1d7f8-tls-certs\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.042405 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.042411 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-dshm\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.042664 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.042542 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-model-cache\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.042664 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.042585 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-home\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.042664 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.042606 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8h5p\" (UniqueName: \"kubernetes.io/projected/d6e90cb3-0324-4828-be59-44040eb1d7f8-kube-api-access-b8h5p\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.042664 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.042622 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-tmp-dir\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.137735 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.137698 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn"] Nov 03 03:20:50.143034 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143007 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-model-cache\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143131 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143070 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-home\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143131 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143103 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8h5p\" (UniqueName: \"kubernetes.io/projected/d6e90cb3-0324-4828-be59-44040eb1d7f8-kube-api-access-b8h5p\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143249 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143144 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-tmp-dir\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143249 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143229 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e90cb3-0324-4828-be59-44040eb1d7f8-tls-certs\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143367 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143266 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-dshm\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143538 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-home\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143648 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143606 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:20:50.143703 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143679 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-tmp-dir\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.143703 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143690 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:20:50.143805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143731 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:20:50.143996 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.143974 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-model-cache\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.145870 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.145845 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-dshm\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.146163 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.146144 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e90cb3-0324-4828-be59-44040eb1d7f8-tls-certs\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.172052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.172011 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" event={"ID":"c861579f-cd2c-4c46-9e25-ea2e49913e8e","Type":"ContainerStarted","Data":"0e64b87d4c341f792bca4a6c3342090f2388342b58823ec781ffbfc39003a17e"} Nov 03 03:20:50.207071 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.207037 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8h5p\" (UniqueName: \"kubernetes.io/projected/d6e90cb3-0324-4828-be59-44040eb1d7f8-kube-api-access-b8h5p\") pod \"e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.275093 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.274990 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:50.671673 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:50.671648 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn"] Nov 03 03:20:50.673646 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:20:50.673619 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6e90cb3_0324_4828_be59_44040eb1d7f8.slice/crio-57848a955081fb0bd8c033b439c0709ab058901f687e77a39d5c92f3a8d2052e WatchSource:0}: Error finding container 57848a955081fb0bd8c033b439c0709ab058901f687e77a39d5c92f3a8d2052e: Status 404 returned error can't find the container with id 57848a955081fb0bd8c033b439c0709ab058901f687e77a39d5c92f3a8d2052e Nov 03 03:20:51.179737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.179690 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" event={"ID":"c861579f-cd2c-4c46-9e25-ea2e49913e8e","Type":"ContainerStarted","Data":"ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a"} Nov 03 03:20:51.181397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.181373 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" event={"ID":"d6e90cb3-0324-4828-be59-44040eb1d7f8","Type":"ContainerStarted","Data":"4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a"} Nov 03 03:20:51.181538 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.181404 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" event={"ID":"d6e90cb3-0324-4828-be59-44040eb1d7f8","Type":"ContainerStarted","Data":"57848a955081fb0bd8c033b439c0709ab058901f687e77a39d5c92f3a8d2052e"} Nov 03 03:20:51.277485 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.275138 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:20:51.334892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.323950 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podStartSLOduration=2.323929518 podStartE2EDuration="2.323929518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:51.304000153 +0000 UTC m=+1186.334598691" watchObservedRunningTime="2026-11-03 03:20:51.323929518 +0000 UTC m=+1186.354528058" Nov 03 03:20:51.430524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.427235 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" podStartSLOduration=2.427211111 podStartE2EDuration="2.427211111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:51.423885285 +0000 UTC m=+1186.454483824" watchObservedRunningTime="2026-11-03 03:20:51.427211111 +0000 UTC m=+1186.457809650" Nov 03 03:20:51.713535 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.710533 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:51.713535 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.710707 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 03:20:51.713535 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:51.710751 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 03:20:52.711473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:52.708850 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 03:20:52.711473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:52.708915 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 03:20:53.714472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:53.713801 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 03:20:53.714472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:53.713865 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 03:20:54.711297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:54.710840 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 03:20:54.711297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:54.710906 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 03:20:55.718470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:55.708880 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 03:20:55.718470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:55.708952 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 03:20:56.709032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:56.708989 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 03:20:56.709242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:56.709056 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 03:20:57.708672 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:57.708633 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 03:20:57.709099 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:57.708704 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 03:20:58.758496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:58.758437 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:58.759129 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:58.759097 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:20:58.759838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:20:58.759820 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:21:00.010077 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:21:00.010044 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:21:00.276228 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:00.276142 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:21:00.288708 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:00.288686 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:21:01.256825 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:01.256797 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:21:04.505287 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.505250 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf"] Nov 03 03:21:04.511519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.511489 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.568563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568532 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.568732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568576 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.568732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568595 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.568732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568682 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.568922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568793 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjvd\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-kube-api-access-2tjvd\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.568922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568841 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.568922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568904 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.569079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568945 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.569079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.568979 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.621798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.621767 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf"] Nov 03 03:21:04.670090 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670061 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670099 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670115 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670153 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670176 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjvd\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-kube-api-access-2tjvd\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670206 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670240 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670273 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670319 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670646 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670740 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670723 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670789 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670748 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670880 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670860 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.670972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.670954 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.672538 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.672515 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.672748 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.672731 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.732116 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.732092 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.732252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.732211 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjvd\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-kube-api-access-2tjvd\") pod \"maas-default-gateway-openshift-default-76f4876667-d7cbf\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:04.826324 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:04.826293 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:05.224581 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:05.224507 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:21:05.229084 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:05.229061 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:21:05.255368 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:05.255343 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf"] Nov 03 03:21:05.256948 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:21:05.256916 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad92d0c_e8da_4e5a_90db_ea62ccd20022.slice/crio-7b0f68e3ff34834dac30d4028239e76ae512cc3175f1daf3c08857fefec012d9 WatchSource:0}: Error finding container 7b0f68e3ff34834dac30d4028239e76ae512cc3175f1daf3c08857fefec012d9: Status 404 returned error can't find the container with id 7b0f68e3ff34834dac30d4028239e76ae512cc3175f1daf3c08857fefec012d9 Nov 03 03:21:05.259557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:05.259522 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:21:05.259681 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:05.259605 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:21:05.259681 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:05.259648 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:21:05.268359 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:05.268336 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" event={"ID":"5ad92d0c-e8da-4e5a-90db-ea62ccd20022","Type":"ContainerStarted","Data":"7b0f68e3ff34834dac30d4028239e76ae512cc3175f1daf3c08857fefec012d9"} Nov 03 03:21:06.275572 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:06.275536 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" event={"ID":"5ad92d0c-e8da-4e5a-90db-ea62ccd20022","Type":"ContainerStarted","Data":"b16ae3d2ae8dda2c56fd3766e7eaec7050e675d0bf1956c2f24e3f115d9b14f7"} Nov 03 03:21:06.399233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:06.399178 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podStartSLOduration=2.399164032 podStartE2EDuration="2.399164032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:06.397487406 +0000 UTC m=+1201.428085938" watchObservedRunningTime="2026-11-03 03:21:06.399164032 +0000 UTC m=+1201.429762570" Nov 03 03:21:06.829470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:06.829073 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:06.832021 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:06.831983 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 03:21:06.832234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:06.832070 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 03:21:07.827471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:07.826992 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 03:21:07.827471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:07.827079 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 03:21:08.831486 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:08.827743 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 03:21:08.831486 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:08.827813 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 03:21:09.826873 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:09.826838 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 03:21:09.827066 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:09.826899 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 03:21:10.827276 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:10.827236 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 03:21:10.827716 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:10.827291 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 03:21:11.827387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:11.827347 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 03:21:11.827830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:11.827408 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 03:21:12.827092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:12.827043 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 03:21:12.827273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:12.827160 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 03:21:14.827685 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:14.827645 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:21:14.828199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:14.827731 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:21:15.868614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:15.868578 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-d7cbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:21:15.869092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:15.868646 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:21:16.282098 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:16.282020 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:16.282612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:16.282582 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:16.283326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:16.283303 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:21:25.658181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.658139 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb"] Nov 03 03:21:25.663107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.663086 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713492 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713438 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713492 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713491 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713661 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srjp\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-kube-api-access-6srjp\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713703 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713784 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713722 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713784 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713742 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713856 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713815 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713856 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713845 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.713976 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.713962 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.716506 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.716484 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b1ad4e-openshift-default-dockercfg-m64tn\"" Nov 03 03:21:25.770409 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.770375 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb"] Nov 03 03:21:25.815572 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.815542 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.815785 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.815584 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.815785 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.815737 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.815903 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.815858 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srjp\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-kube-api-access-6srjp\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.815963 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.815911 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.815963 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.815942 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816074 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816020 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816135 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816116 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816220 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816171 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816220 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816205 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816256 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816467 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816514 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.816768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.816626 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.818231 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.818212 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.818475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.818436 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.877897 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.877860 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.878040 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.877917 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srjp\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-kube-api-access-6srjp\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:25.977242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:25.977156 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:26.408412 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:26.408383 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb"] Nov 03 03:21:26.410081 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:21:26.410045 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9dc019_94af_4592_a0a5_3ab0e1682a1e.slice/crio-561bd09b385553060aed1c0a55c70506345d950fc57e086e2b2f7496d5362cf9 WatchSource:0}: Error finding container 561bd09b385553060aed1c0a55c70506345d950fc57e086e2b2f7496d5362cf9: Status 404 returned error can't find the container with id 561bd09b385553060aed1c0a55c70506345d950fc57e086e2b2f7496d5362cf9 Nov 03 03:21:26.412462 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:26.412410 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:21:26.412576 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:26.412510 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:21:26.412576 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:26.412558 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:21:26.432109 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:26.432077 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" event={"ID":"0a9dc019-94af-4592-a0a5-3ab0e1682a1e","Type":"ContainerStarted","Data":"561bd09b385553060aed1c0a55c70506345d950fc57e086e2b2f7496d5362cf9"} Nov 03 03:21:27.438803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:27.438773 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" event={"ID":"0a9dc019-94af-4592-a0a5-3ab0e1682a1e","Type":"ContainerStarted","Data":"ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344"} Nov 03 03:21:27.626266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:27.626210 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podStartSLOduration=2.626195373 podStartE2EDuration="2.626195373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:27.623769218 +0000 UTC m=+1222.654367757" watchObservedRunningTime="2026-11-03 03:21:27.626195373 +0000 UTC m=+1222.656793911" Nov 03 03:21:27.977517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:27.977479 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:27.979202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:27.979178 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 03:21:27.979331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:27.979222 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 03:21:28.977831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:28.977797 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 03:21:28.978222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:28.977849 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 03:21:29.977859 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:29.977823 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 03:21:29.978253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:29.977875 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 03:21:30.035354 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:21:30.035313 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:21:30.457977 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:30.457938 5117 generic.go:356] "Generic (PLEG): container finished" podID="e7c24f96-ab39-4403-9251-27cceb0c2385" containerID="1f8a5d8e8b9e6f4766dc8d09a2a91fa68799b072a53c89b98c98b9bc01c3feac" exitCode=0 Nov 03 03:21:30.458128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:30.458009 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" event={"ID":"e7c24f96-ab39-4403-9251-27cceb0c2385","Type":"ContainerDied","Data":"1f8a5d8e8b9e6f4766dc8d09a2a91fa68799b072a53c89b98c98b9bc01c3feac"} Nov 03 03:21:30.978001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:30.977964 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 03:21:30.978506 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:30.978028 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 03:21:31.652346 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.652316 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:21:31.678668 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.678634 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c24f96-ab39-4403-9251-27cceb0c2385-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e7c24f96-ab39-4403-9251-27cceb0c2385-archives-path\") pod \"e7c24f96-ab39-4403-9251-27cceb0c2385\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " Nov 03 03:21:31.678859 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.678684 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-trusted-ca-bundle\") pod \"e7c24f96-ab39-4403-9251-27cceb0c2385\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " Nov 03 03:21:31.678859 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.678717 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c24f96-ab39-4403-9251-27cceb0c2385-kube-api-access-hd9fk\" (UniqueName: \"kubernetes.io/projected/e7c24f96-ab39-4403-9251-27cceb0c2385-kube-api-access-hd9fk\") pod \"e7c24f96-ab39-4403-9251-27cceb0c2385\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " Nov 03 03:21:31.678859 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.678834 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-service-ca-bundle\") pod \"e7c24f96-ab39-4403-9251-27cceb0c2385\" (UID: \"e7c24f96-ab39-4403-9251-27cceb0c2385\") " Nov 03 03:21:31.679046 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.678974 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c24f96-ab39-4403-9251-27cceb0c2385-archives-path" pod "e7c24f96-ab39-4403-9251-27cceb0c2385" (UID: "e7c24f96-ab39-4403-9251-27cceb0c2385"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:21:31.679216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.679191 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-service-ca-bundle" pod "e7c24f96-ab39-4403-9251-27cceb0c2385" (UID: "e7c24f96-ab39-4403-9251-27cceb0c2385"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:21:31.679216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.679198 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-trusted-ca-bundle" pod "e7c24f96-ab39-4403-9251-27cceb0c2385" (UID: "e7c24f96-ab39-4403-9251-27cceb0c2385"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:21:31.679309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.679214 5117 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e7c24f96-ab39-4403-9251-27cceb0c2385-archives-path\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:21:31.681119 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.681098 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c24f96-ab39-4403-9251-27cceb0c2385-kube-api-access-hd9fk" pod "e7c24f96-ab39-4403-9251-27cceb0c2385" (UID: "e7c24f96-ab39-4403-9251-27cceb0c2385"). InnerVolumeSpecName "kube-api-access-hd9fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:21:31.779999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.779904 5117 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-service-ca-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:21:31.779999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.779934 5117 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c24f96-ab39-4403-9251-27cceb0c2385-trusted-ca-bundle\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:21:31.779999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.779943 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd9fk\" (UniqueName: \"kubernetes.io/projected/e7c24f96-ab39-4403-9251-27cceb0c2385-kube-api-access-hd9fk\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:21:31.978199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.978163 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 03:21:31.978729 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:31.978233 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 03:21:32.470512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:32.470439 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" event={"ID":"e7c24f96-ab39-4403-9251-27cceb0c2385","Type":"ContainerDied","Data":"a12680e31875ba38adc741a1e16426bb8b8cb002eab1c73ab01d5aa8038a9df6"} Nov 03 03:21:32.470512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:32.470510 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bwmn6-ltxbv" Nov 03 03:21:32.470916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:32.470525 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a12680e31875ba38adc741a1e16426bb8b8cb002eab1c73ab01d5aa8038a9df6" Nov 03 03:21:32.978161 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:32.978117 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 03:21:32.978334 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:32.978191 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 03:21:33.986261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:33.986228 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:33.986740 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:33.986601 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:33.987238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:33.987219 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:21:37.076422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.076387 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq"] Nov 03 03:21:37.076934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.076919 5117 cpu_manager.go:414] "containerMap: removing container" podUID="e7c24f96-ab39-4403-9251-27cceb0c2385" containerName="insights-gathering" Nov 03 03:21:37.076980 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.076936 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c24f96-ab39-4403-9251-27cceb0c2385" containerName="insights-gathering" Nov 03 03:21:37.077038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.077029 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c24f96-ab39-4403-9251-27cceb0c2385" containerName="insights-gathering" Nov 03 03:21:37.081439 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.081422 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.132999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.132960 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tmp-dir\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.133218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.133081 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6z8\" (UniqueName: \"kubernetes.io/projected/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-kube-api-access-nx6z8\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.133218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.133164 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tls-certs\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.133218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.133195 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-dshm\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.133436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.133287 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-model-cache\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.133436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.133348 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-home\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.135753 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.135729 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"openshift-service-ca.crt\"" Nov 03 03:21:37.153828 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.153799 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"kube-root-ca.crt\"" Nov 03 03:21:37.186816 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.186778 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"distinct-w3-68b670-kserve-self-signed-certs\"" Nov 03 03:21:37.186994 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.186826 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"default-dockercfg-x2m5k\"" Nov 03 03:21:37.189278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.189253 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq"] Nov 03 03:21:37.234228 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234192 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-model-cache\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.234418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234352 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-home\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.234418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234377 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tmp-dir\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.234581 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234426 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6z8\" (UniqueName: \"kubernetes.io/projected/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-kube-api-access-nx6z8\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.234581 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234506 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tls-certs\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.234581 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234540 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-dshm\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.234745 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234697 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-model-cache\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.234814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.234759 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-home\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.235031 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.235008 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tmp-dir\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.237191 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.237162 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-dshm\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.237407 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.237388 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tls-certs\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.296326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.296293 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6z8\" (UniqueName: \"kubernetes.io/projected/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-kube-api-access-nx6z8\") pod \"distinct-w3-68b670-kserve-54f58dc987-zvfmq\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.393930 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.393817 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:37.776746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:37.776716 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq"] Nov 03 03:21:37.777574 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:21:37.777542 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f9c45e3_2dec_44ef_b877_0eadbf7c7e4f.slice/crio-a54de191549392d12e6b4f1ce16a6ce9b314aac79c15e915d7f2406db3e2b6d0 WatchSource:0}: Error finding container a54de191549392d12e6b4f1ce16a6ce9b314aac79c15e915d7f2406db3e2b6d0: Status 404 returned error can't find the container with id a54de191549392d12e6b4f1ce16a6ce9b314aac79c15e915d7f2406db3e2b6d0 Nov 03 03:21:38.511881 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:38.511840 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" event={"ID":"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f","Type":"ContainerStarted","Data":"d7e2c6c780a7f4251a421fbac5eba26aa70bbd12068e031ae0fb81bbd519b948"} Nov 03 03:21:38.511881 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:38.511885 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" event={"ID":"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f","Type":"ContainerStarted","Data":"a54de191549392d12e6b4f1ce16a6ce9b314aac79c15e915d7f2406db3e2b6d0"} Nov 03 03:21:38.631668 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:38.631617 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" podStartSLOduration=1.631601992 podStartE2EDuration="1.631601992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:38.630608225 +0000 UTC m=+1233.661206764" watchObservedRunningTime="2026-11-03 03:21:38.631601992 +0000 UTC m=+1233.662200530" Nov 03 03:21:39.394864 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:39.394820 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:47.394865 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:47.394830 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:47.407752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:47.407727 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:21:47.577138 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:21:47.577109 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:22:00.101283 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:22:00.101238 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:22:00.875623 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:00.875576 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:22:00.875985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:00.875952 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:22:10.674947 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.674907 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22"] Nov 03 03:22:10.680670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.680649 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.734104 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.734077 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5a6246-openshift-default-dockercfg-nmbkv\"" Nov 03 03:22:10.787427 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.787393 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22"] Nov 03 03:22:10.802949 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.802913 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.802957 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.803041 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.803073 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803562 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.803541 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803716 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.803623 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803716 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.803655 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9qz\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-kube-api-access-fk9qz\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.803739 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b351c41d-0767-47cc-9221-fa0ea76bfe90-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.803847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.803791 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905009 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.904972 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905156 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905017 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905156 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905141 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905190 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905214 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905265 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905429 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905308 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905429 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905331 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9qz\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-kube-api-access-fk9qz\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905429 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905379 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b351c41d-0767-47cc-9221-fa0ea76bfe90-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905591 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905707 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905684 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905765 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905748 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.905984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.905827 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.906052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.906034 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b351c41d-0767-47cc-9221-fa0ea76bfe90-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.907569 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.907550 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.907760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.907741 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.968263 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.968197 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.968363 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.968310 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9qz\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-kube-api-access-fk9qz\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:10.994857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:10.994824 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:11.427965 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.427933 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22"] Nov 03 03:22:11.432165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.432121 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:22:11.432252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.432216 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:22:11.432291 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.432269 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:22:11.755332 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.755233 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" event={"ID":"b351c41d-0767-47cc-9221-fa0ea76bfe90","Type":"ContainerStarted","Data":"873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48"} Nov 03 03:22:11.755332 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.755316 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" event={"ID":"b351c41d-0767-47cc-9221-fa0ea76bfe90","Type":"ContainerStarted","Data":"73eff9e8442f3d98fe6c61c295e383c22a9249819cb4c04fe2f85003894f82e0"} Nov 03 03:22:11.881629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.881569 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podStartSLOduration=1.881552005 podStartE2EDuration="1.881552005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:22:11.8792833 +0000 UTC m=+1266.909881839" watchObservedRunningTime="2026-11-03 03:22:11.881552005 +0000 UTC m=+1266.912150543" Nov 03 03:22:11.995435 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:11.995402 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:12.996822 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:12.996789 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 03:22:12.997236 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:12.996851 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 03:22:13.995383 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:13.995347 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 03:22:13.995582 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:13.995410 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 03:22:14.995908 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:14.995870 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 03:22:14.996305 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:14.995927 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 03:22:15.995637 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:15.995605 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 03:22:15.995897 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:15.995657 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 03:22:16.995776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:16.995729 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 03:22:16.996322 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:16.995804 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 03:22:17.995724 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:17.995682 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 03:22:17.995941 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:17.995760 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 03:22:19.297157 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:19.297123 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:19.298058 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:19.298039 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:19.299136 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:19.299120 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:22:30.010896 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:22:30.010858 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:22:57.874107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:57.874030 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf"] Nov 03 03:22:57.877783 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:57.877766 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:57.984324 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:57.984289 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"openshift-service-ca.crt\"" Nov 03 03:22:57.984548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:57.984410 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"kube-root-ca.crt\"" Nov 03 03:22:57.984548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:57.984410 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"default-dockercfg-fqsgm\"" Nov 03 03:22:57.984691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:57.984665 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"distinct-w2-de27f5-kserve-self-signed-certs\"" Nov 03 03:22:57.986968 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:57.986941 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf"] Nov 03 03:22:58.013977 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.013943 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-dshm\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.014100 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.013982 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-home\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.014100 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.014062 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nnm8\" (UniqueName: \"kubernetes.io/projected/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-kube-api-access-8nnm8\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.014100 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.014084 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tls-certs\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.014268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.014181 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tmp-dir\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.014268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.014217 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-model-cache\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115209 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-dshm\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115255 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-home\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115309 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nnm8\" (UniqueName: \"kubernetes.io/projected/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-kube-api-access-8nnm8\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115329 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tls-certs\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115352 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tmp-dir\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115368 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-model-cache\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115757 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-home\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115841 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115794 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-model-cache\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.115841 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.115821 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tmp-dir\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.117717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.117697 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-dshm\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.117996 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.117978 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tls-certs\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.177705 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.177622 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nnm8\" (UniqueName: \"kubernetes.io/projected/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-kube-api-access-8nnm8\") pod \"distinct-w2-de27f5-kserve-56b4df96bb-ddlrf\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.190110 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.190082 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:58.589202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:58.589176 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf"] Nov 03 03:22:58.590523 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:22:58.590497 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6eb7f13_1d86_4d51_bc3b_25f3b10fe875.slice/crio-3f4706a0cbd211a424a98ad411c496a704a97ef3fbacddeb63c83c44fc13887d WatchSource:0}: Error finding container 3f4706a0cbd211a424a98ad411c496a704a97ef3fbacddeb63c83c44fc13887d: Status 404 returned error can't find the container with id 3f4706a0cbd211a424a98ad411c496a704a97ef3fbacddeb63c83c44fc13887d Nov 03 03:22:59.112631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:59.112589 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" event={"ID":"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875","Type":"ContainerStarted","Data":"25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb"} Nov 03 03:22:59.113118 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:59.112642 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" event={"ID":"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875","Type":"ContainerStarted","Data":"3f4706a0cbd211a424a98ad411c496a704a97ef3fbacddeb63c83c44fc13887d"} Nov 03 03:22:59.191071 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:59.191041 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:22:59.280695 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:22:59.280628 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podStartSLOduration=2.280608437 podStartE2EDuration="2.280608437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:22:59.279582607 +0000 UTC m=+1314.310181146" watchObservedRunningTime="2026-11-03 03:22:59.280608437 +0000 UTC m=+1314.311206976" Nov 03 03:23:00.011579 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:23:00.011540 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:23:08.190782 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:08.190737 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:23:08.204023 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:08.203995 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:23:09.179669 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:09.179636 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:23:19.885373 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.885334 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt"] Nov 03 03:23:19.889174 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.889154 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939400 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939463 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939715 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939579 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939715 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939607 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939715 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939652 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cecc35b-3e63-46f1-b3e9-2def6166547a-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939876 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939770 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939876 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939826 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939931 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.939986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.939963 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxc2\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-kube-api-access-vcxc2\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:19.942708 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:19.942684 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-68b670-openshift-default-dockercfg-tnxvt\"" Nov 03 03:23:20.003283 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.003249 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt"] Nov 03 03:23:20.041156 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041125 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041344 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041183 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041344 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041298 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxc2\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-kube-api-access-vcxc2\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041505 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041430 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041505 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041498 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041638 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041544 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041698 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041667 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041713 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cecc35b-3e63-46f1-b3e9-2def6166547a-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041784 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041772 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041881 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041856 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.041939 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.041887 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.042150 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.042129 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.042508 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.042485 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cecc35b-3e63-46f1-b3e9-2def6166547a-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.042566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.042544 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.043707 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.043691 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.044206 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.044190 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.103512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.103481 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.103792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.103767 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxc2\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-kube-api-access-vcxc2\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.202126 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.202035 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:20.635651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.635618 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt"] Nov 03 03:23:20.636777 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:23:20.636747 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cecc35b_3e63_46f1_b3e9_2def6166547a.slice/crio-31ad88367df2f006d45ed7756632a018c46926c5d79cbd9298de3a246b64e24e WatchSource:0}: Error finding container 31ad88367df2f006d45ed7756632a018c46926c5d79cbd9298de3a246b64e24e: Status 404 returned error can't find the container with id 31ad88367df2f006d45ed7756632a018c46926c5d79cbd9298de3a246b64e24e Nov 03 03:23:20.639253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.639216 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:23:20.639345 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.639296 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:23:20.639384 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:20.639364 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:23:21.238929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:21.238887 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" event={"ID":"6cecc35b-3e63-46f1-b3e9-2def6166547a","Type":"ContainerStarted","Data":"c5a1106ba3948367f603db69355bbcf9c3d3fa84ca283a6d8d643c0eaae9e375"} Nov 03 03:23:21.239414 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:21.238945 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" event={"ID":"6cecc35b-3e63-46f1-b3e9-2def6166547a","Type":"ContainerStarted","Data":"31ad88367df2f006d45ed7756632a018c46926c5d79cbd9298de3a246b64e24e"} Nov 03 03:23:21.363697 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:21.363645 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podStartSLOduration=2.363631196 podStartE2EDuration="2.363631196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:23:21.361229132 +0000 UTC m=+1336.391827671" watchObservedRunningTime="2026-11-03 03:23:21.363631196 +0000 UTC m=+1336.394229735" Nov 03 03:23:22.210898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:22.210868 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:22.223885 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:22.223558 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 03:23:22.223885 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:22.223614 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 03:23:23.215471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:23.204799 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 03:23:23.215471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:23.204866 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 03:23:24.206909 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:24.206867 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 03:23:24.207201 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:24.207165 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 03:23:25.222475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:25.213573 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 03:23:25.222475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:25.213644 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 03:23:26.205906 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:26.205866 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 03:23:26.206342 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:26.206309 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 03:23:27.202771 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:27.202735 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 03:23:27.203205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:27.202787 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 03:23:28.206527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:28.206492 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:28.206965 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:28.206912 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:28.207688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:28.207670 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:23:30.011298 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:23:30.011258 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:23:41.611003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:41.610966 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn"] Nov 03 03:23:56.632508 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:56.632463 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" podUID="d6e90cb3-0324-4828-be59-44040eb1d7f8" containerName="main" containerID="cri-o://4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a" gracePeriod=45 Nov 03 03:23:56.907257 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:56.907224 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:23:57.064206 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064177 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-tmp-dir\") pod \"d6e90cb3-0324-4828-be59-44040eb1d7f8\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " Nov 03 03:23:57.064397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064218 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e90cb3-0324-4828-be59-44040eb1d7f8-kube-api-access-b8h5p\" (UniqueName: \"kubernetes.io/projected/d6e90cb3-0324-4828-be59-44040eb1d7f8-kube-api-access-b8h5p\") pod \"d6e90cb3-0324-4828-be59-44040eb1d7f8\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " Nov 03 03:23:57.064397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064251 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e90cb3-0324-4828-be59-44040eb1d7f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e90cb3-0324-4828-be59-44040eb1d7f8-tls-certs\") pod \"d6e90cb3-0324-4828-be59-44040eb1d7f8\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " Nov 03 03:23:57.064397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064310 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-home\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-home\") pod \"d6e90cb3-0324-4828-be59-44040eb1d7f8\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " Nov 03 03:23:57.064610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064472 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-dshm\") pod \"d6e90cb3-0324-4828-be59-44040eb1d7f8\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " Nov 03 03:23:57.064610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064504 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-tmp-dir" pod "d6e90cb3-0324-4828-be59-44040eb1d7f8" (UID: "d6e90cb3-0324-4828-be59-44040eb1d7f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:57.064610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064513 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-model-cache\") pod \"d6e90cb3-0324-4828-be59-44040eb1d7f8\" (UID: \"d6e90cb3-0324-4828-be59-44040eb1d7f8\") " Nov 03 03:23:57.064838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064813 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-home" pod "d6e90cb3-0324-4828-be59-44040eb1d7f8" (UID: "d6e90cb3-0324-4828-be59-44040eb1d7f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:57.064935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.064881 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:57.065266 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.065078 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-model-cache" pod "d6e90cb3-0324-4828-be59-44040eb1d7f8" (UID: "d6e90cb3-0324-4828-be59-44040eb1d7f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:57.067391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.067363 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e90cb3-0324-4828-be59-44040eb1d7f8-kube-api-access-b8h5p" pod "d6e90cb3-0324-4828-be59-44040eb1d7f8" (UID: "d6e90cb3-0324-4828-be59-44040eb1d7f8"). InnerVolumeSpecName "kube-api-access-b8h5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:23:57.067525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.067416 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-dshm" pod "d6e90cb3-0324-4828-be59-44040eb1d7f8" (UID: "d6e90cb3-0324-4828-be59-44040eb1d7f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:23:57.067525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.067418 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e90cb3-0324-4828-be59-44040eb1d7f8-tls-certs" pod "d6e90cb3-0324-4828-be59-44040eb1d7f8" (UID: "d6e90cb3-0324-4828-be59-44040eb1d7f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:23:57.165911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.165848 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8h5p\" (UniqueName: \"kubernetes.io/projected/d6e90cb3-0324-4828-be59-44040eb1d7f8-kube-api-access-b8h5p\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:57.165911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.165876 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e90cb3-0324-4828-be59-44040eb1d7f8-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:57.165911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.165890 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:57.165911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.165899 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:57.165911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.165907 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e90cb3-0324-4828-be59-44040eb1d7f8-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:23:57.452364 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.452284 5117 generic.go:356] "Generic (PLEG): container finished" podID="d6e90cb3-0324-4828-be59-44040eb1d7f8" containerID="4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a" exitCode=0 Nov 03 03:23:57.452364 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.452346 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" Nov 03 03:23:57.452626 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.452372 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" event={"ID":"d6e90cb3-0324-4828-be59-44040eb1d7f8","Type":"ContainerDied","Data":"4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a"} Nov 03 03:23:57.452626 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.452422 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn" event={"ID":"d6e90cb3-0324-4828-be59-44040eb1d7f8","Type":"ContainerDied","Data":"57848a955081fb0bd8c033b439c0709ab058901f687e77a39d5c92f3a8d2052e"} Nov 03 03:23:57.452626 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.452471 5117 scope.go:122] "RemoveContainer" containerID="4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a" Nov 03 03:23:57.461858 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.461839 5117 scope.go:122] "RemoveContainer" containerID="4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a" Nov 03 03:23:57.462122 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:23:57.462103 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a\": container with ID starting with 4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a not found: ID does not exist" containerID="4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a" Nov 03 03:23:57.462190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.462135 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a"} err="failed to get container status \"4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a\": rpc error: code = NotFound desc = could not find container \"4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a\": container with ID starting with 4d937a1957bdb462add576028e7631083d737c8003d1496f35ebb0a3db18bc6a not found: ID does not exist" Nov 03 03:23:57.619843 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.619815 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn"] Nov 03 03:23:57.669977 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:57.669952 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-a-e5f308a1-kserve-f74dfbb6-xcvpn"] Nov 03 03:23:59.089668 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:23:59.089636 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e90cb3-0324-4828-be59-44040eb1d7f8" path="/var/lib/kubelet/pods/d6e90cb3-0324-4828-be59-44040eb1d7f8/volumes" Nov 03 03:24:00.011576 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:24:00.011542 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:24:30.011053 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:24:30.010962 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:24:33.223828 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.223796 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd"] Nov 03 03:24:33.224514 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.224493 5117 cpu_manager.go:414] "containerMap: removing container" podUID="d6e90cb3-0324-4828-be59-44040eb1d7f8" containerName="main" Nov 03 03:24:33.224619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.224516 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e90cb3-0324-4828-be59-44040eb1d7f8" containerName="main" Nov 03 03:24:33.224675 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.224656 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e90cb3-0324-4828-be59-44040eb1d7f8" containerName="main" Nov 03 03:24:33.228013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.227993 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.333013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.332985 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-de27f5\"/\"distinct-2-w2-de27f5-kserve-self-signed-certs\"" Nov 03 03:24:33.336063 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.336037 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd"] Nov 03 03:24:33.355970 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.355946 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-model-cache\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.356095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.355976 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkn2s\" (UniqueName: \"kubernetes.io/projected/f23474ee-259b-4255-9c90-0197447f3c61-kube-api-access-fkn2s\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.356147 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.356112 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23474ee-259b-4255-9c90-0197447f3c61-tls-certs\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.356184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.356162 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-dshm\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.356247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.356235 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-tmp-dir\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.356295 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.356284 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-home\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.457839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.457793 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-model-cache\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.457839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.457832 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkn2s\" (UniqueName: \"kubernetes.io/projected/f23474ee-259b-4255-9c90-0197447f3c61-kube-api-access-fkn2s\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.458103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.457900 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23474ee-259b-4255-9c90-0197447f3c61-tls-certs\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.458103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.457923 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-dshm\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.458103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.457961 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-tmp-dir\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.458103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.458087 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-home\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.458312 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.458225 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-model-cache\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.458372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.458329 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-tmp-dir\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.458427 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.458399 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-home\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.460333 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.460306 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-dshm\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.460743 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.460723 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23474ee-259b-4255-9c90-0197447f3c61-tls-certs\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.520710 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.520645 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkn2s\" (UniqueName: \"kubernetes.io/projected/f23474ee-259b-4255-9c90-0197447f3c61-kube-api-access-fkn2s\") pod \"distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.549661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.549630 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:33.944763 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:33.944734 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd"] Nov 03 03:24:33.945722 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:24:33.945696 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf23474ee_259b_4255_9c90_0197447f3c61.slice/crio-24195fc357d4d9dceb4019b7cd6453cf5ead0b169a036e32fb0d352b7efaf6f8 WatchSource:0}: Error finding container 24195fc357d4d9dceb4019b7cd6453cf5ead0b169a036e32fb0d352b7efaf6f8: Status 404 returned error can't find the container with id 24195fc357d4d9dceb4019b7cd6453cf5ead0b169a036e32fb0d352b7efaf6f8 Nov 03 03:24:34.699774 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:34.699737 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" event={"ID":"f23474ee-259b-4255-9c90-0197447f3c61","Type":"ContainerStarted","Data":"86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355"} Nov 03 03:24:34.700164 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:34.699788 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" event={"ID":"f23474ee-259b-4255-9c90-0197447f3c61","Type":"ContainerStarted","Data":"24195fc357d4d9dceb4019b7cd6453cf5ead0b169a036e32fb0d352b7efaf6f8"} Nov 03 03:24:34.818727 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:34.818636 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podStartSLOduration=1.818615477 podStartE2EDuration="1.818615477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:34.818545686 +0000 UTC m=+1409.849144224" watchObservedRunningTime="2026-11-03 03:24:34.818615477 +0000 UTC m=+1409.849214016" Nov 03 03:24:35.470927 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.470884 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb"] Nov 03 03:24:35.474837 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.474813 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.523409 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.523364 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn"] Nov 03 03:24:35.528629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.528602 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.550288 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.550255 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:35.577865 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.577828 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb"] Nov 03 03:24:35.580818 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.580788 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.580951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.580836 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.580951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.580879 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.581041 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.580994 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.581101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.581072 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.581152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.581141 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.581197 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.581169 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjcjc\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-kube-api-access-sjcjc\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.581400 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.581378 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.581439 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.581422 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2949fa5-406f-4816-af0a-c93db25f9db5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.668434 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.652931 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn"] Nov 03 03:24:35.682152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682123 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682317 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682166 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682317 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682204 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682317 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682222 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682514 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682366 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682514 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682408 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682514 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682459 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f18784-21f2-4fa4-b018-4bf4a5de259f-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682514 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682487 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682518 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682550 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682578 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682624 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjcjc\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-kube-api-access-sjcjc\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682697 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682743 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682796 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5fhp\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-kube-api-access-w5fhp\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682809 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682866 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.682960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682871 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.682960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.682906 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.683278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.683196 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.683341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.683273 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2949fa5-406f-4816-af0a-c93db25f9db5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.683500 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.683473 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.684049 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.684024 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2949fa5-406f-4816-af0a-c93db25f9db5-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.685725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.685701 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.685845 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.685705 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.772357 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.772277 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjcjc\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-kube-api-access-sjcjc\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.773995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.773973 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.784839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.784805 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5fhp\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-kube-api-access-w5fhp\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.784973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.784861 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.784999 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785036 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785074 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f18784-21f2-4fa4-b018-4bf4a5de259f-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785230 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785200 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785246 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785357 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785329 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785411 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785374 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785479 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785589 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785687 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785751 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.785885 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.785864 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f18784-21f2-4fa4-b018-4bf4a5de259f-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.787408 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.787388 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.788335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.788311 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.792154 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.792125 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:35.863470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.855009 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:35.863470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:35.855133 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5fhp\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-kube-api-access-w5fhp\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:36.148225 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.148183 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:36.268157 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.267921 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb"] Nov 03 03:24:36.272782 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:24:36.272492 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2949fa5_406f_4816_af0a_c93db25f9db5.slice/crio-cdb15beae434a3f7358329445e4f49c8b14a0daa259819d836a8ca2f0c92ac89 WatchSource:0}: Error finding container cdb15beae434a3f7358329445e4f49c8b14a0daa259819d836a8ca2f0c92ac89: Status 404 returned error can't find the container with id cdb15beae434a3f7358329445e4f49c8b14a0daa259819d836a8ca2f0c92ac89 Nov 03 03:24:36.275844 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.275804 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:36.275935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.275898 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:36.275975 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.275940 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:36.550933 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.550892 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn"] Nov 03 03:24:36.559628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.557927 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:36.559628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.558012 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:36.559628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.558058 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:36.723393 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.722733 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" event={"ID":"b2949fa5-406f-4816-af0a-c93db25f9db5","Type":"ContainerStarted","Data":"6b27161bfbd0a3be1051379ff91b16c0d2a96a3cb901bb07bd25c29d9bca4109"} Nov 03 03:24:36.723393 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.722787 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" event={"ID":"b2949fa5-406f-4816-af0a-c93db25f9db5","Type":"ContainerStarted","Data":"cdb15beae434a3f7358329445e4f49c8b14a0daa259819d836a8ca2f0c92ac89"} Nov 03 03:24:36.725057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.725026 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" event={"ID":"65f18784-21f2-4fa4-b018-4bf4a5de259f","Type":"ContainerStarted","Data":"4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e"} Nov 03 03:24:36.725212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.725099 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" event={"ID":"65f18784-21f2-4fa4-b018-4bf4a5de259f","Type":"ContainerStarted","Data":"c3f68b1230df44de0bb93a62685fdce499badd5439a86963fb76c8b0113de0cf"} Nov 03 03:24:36.793213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.793179 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:36.847899 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.847826 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podStartSLOduration=1.847806405 podStartE2EDuration="1.847806405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:36.84727261 +0000 UTC m=+1411.877871149" watchObservedRunningTime="2026-11-03 03:24:36.847806405 +0000 UTC m=+1411.878404945" Nov 03 03:24:36.972250 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:36.972194 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podStartSLOduration=1.97217756 podStartE2EDuration="1.97217756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:36.969395882 +0000 UTC m=+1411.999994421" watchObservedRunningTime="2026-11-03 03:24:36.97217756 +0000 UTC m=+1412.002776098" Nov 03 03:24:37.153617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:37.152750 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:37.153943 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:37.153916 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 03:24:37.154092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:37.154069 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 03:24:37.794972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:37.794937 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 03:24:37.795411 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:37.794997 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 03:24:38.153472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:38.152884 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 03:24:38.153472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:38.152950 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 03:24:38.793149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:38.793118 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 03:24:38.793352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:38.793171 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 03:24:39.149474 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:39.148874 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 03:24:39.149474 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:39.148972 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 03:24:39.792703 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:39.792667 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 03:24:39.792871 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:39.792736 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 03:24:40.150474 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:40.149865 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 03:24:40.150474 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:40.149931 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 03:24:40.792849 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:40.792807 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 03:24:40.793057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:40.792866 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 03:24:41.148979 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:41.148906 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 03:24:41.149207 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:41.148997 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 03:24:41.793364 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:41.793329 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 03:24:41.793823 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:41.793387 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 03:24:41.871491 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:41.871460 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954"] Nov 03 03:24:41.875475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:41.875439 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:41.973544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:41.973508 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954"] Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029288 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029344 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029426 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029496 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ml5\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-kube-api-access-v6ml5\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029582 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029640 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029671 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029706 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.030482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.029750 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.138108 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.138063 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.139391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.139348 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.149093 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.138901 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149328 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149576 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149625 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149680 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149697 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149728 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149820 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.149872 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ml5\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-kube-api-access-v6ml5\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.150209 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.150272 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.150948 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.150823 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 03:24:42.150948 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.150878 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 03:24:42.151315 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.151237 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.152882 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.152841 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.153109 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.153085 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.246595 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.246178 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ml5\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-kube-api-access-v6ml5\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.250958 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.250152 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.489998 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.489910 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:42.792895 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.792806 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 03:24:42.792895 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.792881 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 03:24:42.951484 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.951439 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954"] Nov 03 03:24:42.952931 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:24:42.952903 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031c117b_cd8c_4bc5_84b8_68aea1cf5748.slice/crio-2bf0f1b92758a3e5cd13d24d8ce92368bc27c4a1af5c339a69e629f037291b5a WatchSource:0}: Error finding container 2bf0f1b92758a3e5cd13d24d8ce92368bc27c4a1af5c339a69e629f037291b5a: Status 404 returned error can't find the container with id 2bf0f1b92758a3e5cd13d24d8ce92368bc27c4a1af5c339a69e629f037291b5a Nov 03 03:24:42.955820 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.955780 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:42.955910 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.955888 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:42.955952 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:42.955937 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:24:43.149719 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.149673 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 03:24:43.149908 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.149755 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 03:24:43.551527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.551135 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:43.571472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.567514 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:43.846465 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.845565 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" event={"ID":"031c117b-cd8c-4bc5-84b8-68aea1cf5748","Type":"ContainerStarted","Data":"059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f"} Nov 03 03:24:43.846465 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.845663 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" event={"ID":"031c117b-cd8c-4bc5-84b8-68aea1cf5748","Type":"ContainerStarted","Data":"2bf0f1b92758a3e5cd13d24d8ce92368bc27c4a1af5c339a69e629f037291b5a"} Nov 03 03:24:43.861489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.858852 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:24:43.980125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.980072 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podStartSLOduration=2.9800577 podStartE2EDuration="2.9800577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:43.977659519 +0000 UTC m=+1419.008258057" watchObservedRunningTime="2026-11-03 03:24:43.9800577 +0000 UTC m=+1419.010656238" Nov 03 03:24:43.996603 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.996567 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:43.997792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.997768 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:43.998771 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:43.998748 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 03:24:44.366786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:44.366752 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:44.367462 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:44.367425 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:44.368397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:44.368367 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:24:44.491852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:44.491819 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:44.492755 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:44.492730 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 03:24:44.492881 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:44.492778 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 03:24:45.490930 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:45.490897 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 03:24:45.491358 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:45.490947 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 03:24:46.490742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:46.490711 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 03:24:46.490931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:46.490766 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 03:24:47.490758 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:47.490724 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 03:24:47.491131 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:47.490787 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 03:24:48.491330 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:48.491291 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 03:24:48.491757 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:48.491362 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 03:24:49.490486 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:49.490417 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 03:24:49.490691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:49.490507 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 03:24:50.494913 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:50.494881 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:50.495313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:50.495281 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:24:50.495975 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:24:50.495954 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:25:00.011573 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:25:00.011535 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:25:19.198530 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.198495 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8"] Nov 03 03:25:19.202582 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.202564 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.239717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.239672 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.239914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.239826 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79913da2-9bfb-45da-8c50-7e081c821789-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.239914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.239862 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.240036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.240008 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.240090 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.240047 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67ql\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-kube-api-access-x67ql\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.240139 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.240126 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.240187 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.240164 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79913da2-9bfb-45da-8c50-7e081c821789-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.240226 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.240204 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.240226 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.240220 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.256459 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.256413 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de27f5-openshift-default-dockercfg-djfq7\"" Nov 03 03:25:19.310506 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.310459 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8"] Nov 03 03:25:19.341166 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341125 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341252 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79913da2-9bfb-45da-8c50-7e081c821789-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341279 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341463 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341380 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341660 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341405 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67ql\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-kube-api-access-x67ql\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341713 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341722 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341602 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341932 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341802 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79913da2-9bfb-45da-8c50-7e081c821789-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341932 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341867 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.341932 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.341896 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.342060 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.342040 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.342283 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.342262 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.342790 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.342768 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79913da2-9bfb-45da-8c50-7e081c821789-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.344437 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.344415 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79913da2-9bfb-45da-8c50-7e081c821789-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.344626 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.344605 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.403792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.403742 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.403956 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.403895 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67ql\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-kube-api-access-x67ql\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.516186 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.516086 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:19.949357 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.949329 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8"] Nov 03 03:25:19.950123 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:25:19.950093 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79913da2_9bfb_45da_8c50_7e081c821789.slice/crio-ff560f2460d25e6e07e3a36de54904fb6c51ebfbd06ae8155b66478f91256eaf WatchSource:0}: Error finding container ff560f2460d25e6e07e3a36de54904fb6c51ebfbd06ae8155b66478f91256eaf: Status 404 returned error can't find the container with id ff560f2460d25e6e07e3a36de54904fb6c51ebfbd06ae8155b66478f91256eaf Nov 03 03:25:19.952190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.952163 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:19.952264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.952222 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:19.952264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:19.952253 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:20.123585 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:20.123546 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" event={"ID":"79913da2-9bfb-45da-8c50-7e081c821789","Type":"ContainerStarted","Data":"47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9"} Nov 03 03:25:20.123776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:20.123607 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" event={"ID":"79913da2-9bfb-45da-8c50-7e081c821789","Type":"ContainerStarted","Data":"ff560f2460d25e6e07e3a36de54904fb6c51ebfbd06ae8155b66478f91256eaf"} Nov 03 03:25:20.250210 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:20.250085 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podStartSLOduration=1.25006996 podStartE2EDuration="1.25006996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:20.247273051 +0000 UTC m=+1455.277871603" watchObservedRunningTime="2026-11-03 03:25:20.25006996 +0000 UTC m=+1455.280668497" Nov 03 03:25:20.519530 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:20.518549 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:21.522472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:21.518515 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 03:25:21.522472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:21.518575 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 03:25:22.518466 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:22.517865 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 03:25:22.518466 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:22.517932 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 03:25:23.521468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:23.520822 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 03:25:23.521468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:23.520887 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 03:25:24.524502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:24.523827 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 03:25:24.524502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:24.523889 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 03:25:25.520470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:25.519918 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 03:25:25.520470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:25.519981 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 03:25:26.517015 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:26.516970 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 03:25:26.517477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:26.517040 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 03:25:27.236245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.236210 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9"] Nov 03 03:25:27.240278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.240255 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.331565 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.331533 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9"] Nov 03 03:25:27.356390 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356360 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k622t\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-kube-api-access-k622t\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.356601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356399 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.356601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356553 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.356707 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356672 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.356758 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356706 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.356835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356816 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.356895 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356880 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a34e4e-ac22-4381-8c20-4d86850cebe0-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.356951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.356908 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.357075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.357055 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.458616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.458585 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a34e4e-ac22-4381-8c20-4d86850cebe0-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.458616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.458621 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.458832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.458685 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.458832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.458717 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k622t\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-kube-api-access-k622t\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.458947 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.458861 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459002 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.458955 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459055 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459025 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459110 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459053 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459161 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459120 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459207 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459157 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459384 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459336 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459529 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459533 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a34e4e-ac22-4381-8c20-4d86850cebe0-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.459718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.459549 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.461372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.461350 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.461495 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.461424 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.521401 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.521349 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:27.521819 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.521798 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:27.522631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.522614 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:25:27.525162 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.525130 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.525236 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.525192 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k622t\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-kube-api-access-k622t\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.538673 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.538641 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m"] Nov 03 03:25:27.545900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.545873 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.558986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.558964 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:27.567169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.567140 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv"] Nov 03 03:25:27.571517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.571498 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.600134 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.600107 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m"] Nov 03 03:25:27.635056 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.635016 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv"] Nov 03 03:25:27.660910 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.660877 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661068 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.660937 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661068 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.660967 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661068 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661025 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661071 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661103 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661135 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661153 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661175 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661324 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661257 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661324 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661293 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661324 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661309 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661329 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9597z\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-kube-api-access-9597z\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661365 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhzg\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-kube-api-access-5bhzg\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661383 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661418 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661413 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.661555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661470 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.661555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.661524 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763100 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763067 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763106 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763145 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763173 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9597z\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-kube-api-access-9597z\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763219 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhzg\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-kube-api-access-5bhzg\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763246 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763274 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763319 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763365 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763434 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763479 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763506 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763538 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763586 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763598 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763622 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763668 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763701 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763734 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763767 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.763926 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763917 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.764338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.763932 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.764338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.764116 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.764483 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.764398 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.764542 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.764434 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.764542 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.764499 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.764664 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.764568 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.764759 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.764736 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.766227 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.766202 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.766331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.766314 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.766616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.766595 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.766796 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.766780 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.825661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.825588 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.825848 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.825805 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9597z\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-kube-api-access-9597z\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.826336 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.826308 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.826797 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.826764 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhzg\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-kube-api-access-5bhzg\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:27.860632 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.860603 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:27.894761 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:27.894650 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:28.011956 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:25:28.011915 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a34e4e_ac22_4381_8c20_4d86850cebe0.slice/crio-3d1c36fad63ac21f81f0df82c324467f03331d5046e859e352b661cc2ec801b5 WatchSource:0}: Error finding container 3d1c36fad63ac21f81f0df82c324467f03331d5046e859e352b661cc2ec801b5: Status 404 returned error can't find the container with id 3d1c36fad63ac21f81f0df82c324467f03331d5046e859e352b661cc2ec801b5 Nov 03 03:25:28.017564 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.016285 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.017564 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.016366 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.017564 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.016405 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.017564 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.017039 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9"] Nov 03 03:25:28.175533 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.175486 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" event={"ID":"27a34e4e-ac22-4381-8c20-4d86850cebe0","Type":"ContainerStarted","Data":"3432accbf8ead2cc98912757da4533d04389863fb0a3dba3dd9dacc80434f6ed"} Nov 03 03:25:28.175710 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.175601 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" event={"ID":"27a34e4e-ac22-4381-8c20-4d86850cebe0","Type":"ContainerStarted","Data":"3d1c36fad63ac21f81f0df82c324467f03331d5046e859e352b661cc2ec801b5"} Nov 03 03:25:28.260264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.260236 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m"] Nov 03 03:25:28.261473 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:25:28.261432 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8841cb2_59ba_47ef_921c_9ed8105c99ca.slice/crio-45d89a2c9b790f22a7bfe31b8f0f54a5b772e7f7ffc4e6fe4c8960e44b88ff0a WatchSource:0}: Error finding container 45d89a2c9b790f22a7bfe31b8f0f54a5b772e7f7ffc4e6fe4c8960e44b88ff0a: Status 404 returned error can't find the container with id 45d89a2c9b790f22a7bfe31b8f0f54a5b772e7f7ffc4e6fe4c8960e44b88ff0a Nov 03 03:25:28.264020 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.263986 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.264135 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.264051 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.264135 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.264110 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.340942 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.340376 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podStartSLOduration=1.3403557240000001 podStartE2EDuration="1.340355724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:28.299750754 +0000 UTC m=+1463.330349303" watchObservedRunningTime="2026-11-03 03:25:28.340355724 +0000 UTC m=+1463.370954263" Nov 03 03:25:28.343904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.343877 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv"] Nov 03 03:25:28.344156 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:25:28.344119 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f571655_0c7e_44a5_a8e4_fa4ef0ac09f6.slice/crio-7b3bd738ae0bce57eb3d6b6b4bc7308fb51d77da0f41f41ab6715ff560a3fd40 WatchSource:0}: Error finding container 7b3bd738ae0bce57eb3d6b6b4bc7308fb51d77da0f41f41ab6715ff560a3fd40: Status 404 returned error can't find the container with id 7b3bd738ae0bce57eb3d6b6b4bc7308fb51d77da0f41f41ab6715ff560a3fd40 Nov 03 03:25:28.346595 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.346564 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.346676 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.346623 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.346676 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.346651 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:28.559840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:28.559807 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:29.190814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.190778 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" event={"ID":"c8841cb2-59ba-47ef-921c-9ed8105c99ca","Type":"ContainerStarted","Data":"ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2"} Nov 03 03:25:29.190990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.190833 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" event={"ID":"c8841cb2-59ba-47ef-921c-9ed8105c99ca","Type":"ContainerStarted","Data":"45d89a2c9b790f22a7bfe31b8f0f54a5b772e7f7ffc4e6fe4c8960e44b88ff0a"} Nov 03 03:25:29.193007 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.192978 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" event={"ID":"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6","Type":"ContainerStarted","Data":"ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a"} Nov 03 03:25:29.193121 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.193032 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" event={"ID":"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6","Type":"ContainerStarted","Data":"7b3bd738ae0bce57eb3d6b6b4bc7308fb51d77da0f41f41ab6715ff560a3fd40"} Nov 03 03:25:29.372594 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.372522 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podStartSLOduration=2.372495411 podStartE2EDuration="2.372495411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:29.347923782 +0000 UTC m=+1464.378522321" watchObservedRunningTime="2026-11-03 03:25:29.372495411 +0000 UTC m=+1464.403093950" Nov 03 03:25:29.562380 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.562347 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 03:25:29.562956 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.562927 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 03:25:29.874532 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.871611 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:29.874532 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.873895 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 03:25:29.874532 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.874291 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 03:25:29.894932 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.894896 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:29.896307 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.896277 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:25:29.896427 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:29.896329 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 03:25:30.089796 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:25:30.089750 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:25:30.576473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:30.571726 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 03:25:30.576473 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:30.571803 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 03:25:30.877528 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:30.874293 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 03:25:30.877528 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:30.874354 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 03:25:30.895194 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:30.895164 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:25:30.895367 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:30.895217 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 03:25:31.565478 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:31.564047 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 03:25:31.565478 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:31.564130 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 03:25:31.864920 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:31.864826 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 03:25:31.866270 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:31.866235 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 03:25:31.895489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:31.895419 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:25:31.895734 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:31.895711 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 03:25:32.568524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:32.568484 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 03:25:32.568818 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:32.568788 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 03:25:32.863903 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:32.863815 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 03:25:32.864135 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:32.864108 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 03:25:32.895202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:32.895174 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:25:32.895641 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:32.895529 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 03:25:33.596471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:33.595744 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 03:25:33.596471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:33.595801 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 03:25:33.879523 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:33.863610 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 03:25:33.879523 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:33.863666 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 03:25:33.896472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:33.895184 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:25:33.896472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:33.895251 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 03:25:34.354472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.351230 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podStartSLOduration=7.351209481 podStartE2EDuration="7.351209481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:29.529822106 +0000 UTC m=+1464.560420647" watchObservedRunningTime="2026-11-03 03:25:34.351209481 +0000 UTC m=+1469.381808023" Nov 03 03:25:34.354472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.352811 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t"] Nov 03 03:25:34.372470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.367253 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.478865 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.478322 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t"] Nov 03 03:25:34.541350 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.541309 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.545698 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.545670 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.545896 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.545882 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.546017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.546003 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.546156 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.546142 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.546260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.546248 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjq9g\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-kube-api-access-zjq9g\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.546395 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.546383 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857e8c66-4d5f-4757-b916-056da42d0e3c-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.550576 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.550554 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.550723 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.550707 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.573098 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.572045 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 03:25:34.573098 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.572099 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 03:25:34.652010 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.651922 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857e8c66-4d5f-4757-b916-056da42d0e3c-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652010 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.651993 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652021 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652061 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652114 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652284 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652187 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652284 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652234 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652298 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652352 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjq9g\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-kube-api-access-zjq9g\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.652890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652864 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.653006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.652924 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.653197 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.653175 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.653544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.653519 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.653814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.653574 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857e8c66-4d5f-4757-b916-056da42d0e3c-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.655515 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.655494 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.655739 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.655724 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.715390 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.715354 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjq9g\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-kube-api-access-zjq9g\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.715860 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.715833 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:34.861039 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.861006 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 03:25:34.861216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.861058 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 03:25:34.910944 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.910579 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 03:25:34.910944 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.910683 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 03:25:34.996729 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:34.996688 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:35.499846 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:35.499299 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t"] Nov 03 03:25:35.505606 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:35.505558 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:35.505750 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:35.505646 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:35.505750 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:35.505695 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:25:36.043433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.043370 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:36.044893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.044868 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:36.047185 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.047166 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:25:36.120265 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.120233 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:36.120688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.120663 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:36.121997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.121976 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:25:36.145517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.145487 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:36.146486 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.146118 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:36.147245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.147226 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:25:36.345670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.345625 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" event={"ID":"857e8c66-4d5f-4757-b916-056da42d0e3c","Type":"ContainerStarted","Data":"1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733"} Nov 03 03:25:36.346571 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.346528 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" event={"ID":"857e8c66-4d5f-4757-b916-056da42d0e3c","Type":"ContainerStarted","Data":"1a1ffd4c2f7a0207b31e2bd079ba2ec7922409967e39f2f0e727ef2be035f11e"} Nov 03 03:25:36.691034 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.690937 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podStartSLOduration=2.690922774 podStartE2EDuration="2.690922774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:36.689217097 +0000 UTC m=+1471.719815634" watchObservedRunningTime="2026-11-03 03:25:36.690922774 +0000 UTC m=+1471.721521376" Nov 03 03:25:36.998072 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:36.997984 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:37.024852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:37.024816 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 03:25:37.025044 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:37.024881 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 03:25:37.997654 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:37.997614 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 03:25:37.998052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:37.997690 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 03:25:38.997969 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:38.997928 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 03:25:38.998602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:38.998565 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 03:25:40.001033 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:40.000993 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 03:25:40.001724 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:40.001679 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 03:25:40.170585 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:40.170535 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:40.764602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:40.764551 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:40.891361 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:40.890749 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:25:41.008482 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:41.002748 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 03:25:41.030655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:41.030554 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 03:25:41.997590 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:41.997550 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 03:25:41.997804 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:41.997612 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 03:25:42.997635 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:42.997599 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 03:25:42.998118 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:42.997663 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 03:25:44.497216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:44.497182 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:44.497739 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:44.497725 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:25:44.498659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:25:44.498633 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:26:00.011025 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:26:00.010987 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:26:04.555609 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:04.555536 5117 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.640760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:04.640719 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.761292 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:04.761239 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:04.845673 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:04.845628 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 03:26:05.059500 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.050767 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:05.059500 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.050833 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.258528 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.190598 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.666924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.637703 5117 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Nov 03 03:26:05.667865 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.667823 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.704469 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.674563 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:26:05.776518 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.772525 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 03:26:05.803773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.803742 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:26:05.867795 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.867738 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:05.869563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:05.869519 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:26:06.224230 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:06.224171 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 03:26:06.225415 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:06.225393 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:26:06.676402 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:06.676343 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.814978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:06.814936 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:06.872487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:06.870635 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:07.189026 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:07.188994 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:26:07.217380 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:07.217350 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:26:07.225073 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:07.225032 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:07.297569 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:07.297536 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:26:07.440413 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:07.439728 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 03:26:07.484781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:07.484285 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 03:26:07.706537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:07.705790 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 03:26:19.707665 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.707617 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp"] Nov 03 03:26:19.711497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.711436 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845156 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845411 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845307 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845411 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845354 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845672 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845474 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845672 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845564 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrslx\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-kube-api-access-mrslx\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845672 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845636 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845909 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845703 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.845909 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.845857 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.846026 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.846005 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.929725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.929686 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp"] Nov 03 03:26:19.947540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947486 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.947748 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947571 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrslx\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-kube-api-access-mrslx\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.947748 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947645 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.947748 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947713 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.947925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947757 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.947925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947793 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.947925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947844 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.948086 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947935 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.948086 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947983 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.948086 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.947992 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.948233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.948177 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.948233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.948210 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.948344 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.948283 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.948620 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.948596 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.950552 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.950528 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:19.950715 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:19.950693 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:20.009873 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.009790 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:20.010048 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.010006 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrslx\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-kube-api-access-mrslx\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:20.024865 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.024829 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:20.455761 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.455735 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp"] Nov 03 03:26:20.456819 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:26:20.456761 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cc62d3_2b5f_4f34_a2ba_7648a5094228.slice/crio-bb4cfdc73f6f7fe849a1f6a083620c7f60c6b59c193e1684b80f25fbcb9e083f WatchSource:0}: Error finding container bb4cfdc73f6f7fe849a1f6a083620c7f60c6b59c193e1684b80f25fbcb9e083f: Status 404 returned error can't find the container with id bb4cfdc73f6f7fe849a1f6a083620c7f60c6b59c193e1684b80f25fbcb9e083f Nov 03 03:26:20.459499 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.459439 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:26:20.459621 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.459552 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:26:20.459621 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.459599 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:26:20.874604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.874563 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" event={"ID":"50cc62d3-2b5f-4f34-a2ba-7648a5094228","Type":"ContainerStarted","Data":"c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae"} Nov 03 03:26:20.875079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:20.874632 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" event={"ID":"50cc62d3-2b5f-4f34-a2ba-7648a5094228","Type":"ContainerStarted","Data":"bb4cfdc73f6f7fe849a1f6a083620c7f60c6b59c193e1684b80f25fbcb9e083f"} Nov 03 03:26:21.025781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.025737 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:21.027678 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.027647 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:26:21.027829 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.027700 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 03:26:21.049626 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.049576 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podStartSLOduration=2.049562263 podStartE2EDuration="2.049562263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:21.047396525 +0000 UTC m=+1516.077995076" watchObservedRunningTime="2026-11-03 03:26:21.049562263 +0000 UTC m=+1516.080161029" Nov 03 03:26:21.676461 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.676417 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn"] Nov 03 03:26:21.680574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.680558 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.733857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.733824 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9902a0c2-serving-cert\"" Nov 03 03:26:21.770218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.770186 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-metrics-certs\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.770387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.770234 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdwl\" (UniqueName: \"kubernetes.io/projected/59763795-5ea6-428f-b90a-165ddb42ba90-kube-api-access-mcdwl\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.770467 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.770436 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59763795-5ea6-428f-b90a-165ddb42ba90-openshift-service-ca\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.770512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.770478 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.784013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.783983 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn"] Nov 03 03:26:21.871099 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.871067 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-metrics-certs\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.871258 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.871126 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdwl\" (UniqueName: \"kubernetes.io/projected/59763795-5ea6-428f-b90a-165ddb42ba90-kube-api-access-mcdwl\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.871314 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.871301 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59763795-5ea6-428f-b90a-165ddb42ba90-openshift-service-ca\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.871361 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.871330 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.871606 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:26:21.871582 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-serving-cert: secret "maas-api-e2e-mt-9902a0c2-serving-cert" not found Nov 03 03:26:21.871865 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:26:21.871822 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls podName:59763795-5ea6-428f-b90a-165ddb42ba90 nodeName:}" failed. No retries permitted until 2026-11-03 03:26:22.371801078 +0000 UTC m=+1517.402399612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls") pod "maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" (UID: "59763795-5ea6-428f-b90a-165ddb42ba90") : secret "maas-api-e2e-mt-9902a0c2-serving-cert" not found Nov 03 03:26:21.872137 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.872116 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59763795-5ea6-428f-b90a-165ddb42ba90-openshift-service-ca\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.874017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.873998 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-metrics-certs\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:21.952578 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:21.952437 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdwl\" (UniqueName: \"kubernetes.io/projected/59763795-5ea6-428f-b90a-165ddb42ba90-kube-api-access-mcdwl\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:22.027544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:22.027506 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:26:22.027746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:22.027567 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 03:26:22.378399 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:22.378362 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:22.380912 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:22.380888 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:22.595324 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:22.595287 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:22.996483 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:22.996438 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn"] Nov 03 03:26:22.997826 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:26:22.997805 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59763795_5ea6_428f_b90a_165ddb42ba90.slice/crio-434f09a741840968783019951c59aac7f7ab8e2b1a792b957605725c05b54e11 WatchSource:0}: Error finding container 434f09a741840968783019951c59aac7f7ab8e2b1a792b957605725c05b54e11: Status 404 returned error can't find the container with id 434f09a741840968783019951c59aac7f7ab8e2b1a792b957605725c05b54e11 Nov 03 03:26:22.999809 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:22.999787 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:26:23.030585 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:23.030555 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:26:23.030716 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:23.030622 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 03:26:23.897973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:23.897315 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" event={"ID":"59763795-5ea6-428f-b90a-165ddb42ba90","Type":"ContainerStarted","Data":"434f09a741840968783019951c59aac7f7ab8e2b1a792b957605725c05b54e11"} Nov 03 03:26:24.029471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:24.026046 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:26:24.029471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:24.026104 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 03:26:24.938468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:24.937926 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" event={"ID":"59763795-5ea6-428f-b90a-165ddb42ba90","Type":"ContainerStarted","Data":"d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32"} Nov 03 03:26:24.948674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:24.948638 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:25.030997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:25.030952 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:26:25.031575 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:25.031025 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 03:26:25.070920 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:25.070864 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" podStartSLOduration=3.133840922 podStartE2EDuration="4.070846876s" totalImagesPullingTime="937.005954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:25.06753252 +0000 UTC m=+1520.098131063" watchObservedRunningTime="2026-11-03 03:26:25.070846876 +0000 UTC m=+1520.101445413" Nov 03 03:26:26.026082 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:26.026043 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:26:26.026296 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:26.026103 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 03:26:27.026152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:27.026118 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 03:26:27.026560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:27.026198 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 03:26:28.030616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:28.030587 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:28.031086 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:28.031065 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:28.031930 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:28.031910 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:26:30.011868 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:26:30.011822 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:26:30.950300 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:30.950273 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:26:37.652511 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.652474 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh"] Nov 03 03:26:37.660835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.660806 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.692776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.692740 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-tmp-dir\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.692981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.692873 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-model-cache\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.692981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.692950 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38375563-5dc0-4101-93af-ad9978d8997b-tls-certs\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.693135 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.693112 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-dshm\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.693278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.693258 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpqd\" (UniqueName: \"kubernetes.io/projected/38375563-5dc0-4101-93af-ad9978d8997b-kube-api-access-kqpqd\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.693349 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.693312 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-home\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.712110 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.712075 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh"] Nov 03 03:26:37.759504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.759474 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"openshift-service-ca.crt\"" Nov 03 03:26:37.759504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.759499 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"kube-root-ca.crt\"" Nov 03 03:26:37.759764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.759589 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"distinct-w0-0aa6b9-kserve-self-signed-certs\"" Nov 03 03:26:37.764717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.764690 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"default-dockercfg-twz4w\"" Nov 03 03:26:37.794524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.794491 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpqd\" (UniqueName: \"kubernetes.io/projected/38375563-5dc0-4101-93af-ad9978d8997b-kube-api-access-kqpqd\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.794695 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.794538 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-home\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.794695 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.794648 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-tmp-dir\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.794786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.794704 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-model-cache\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.794786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.794756 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38375563-5dc0-4101-93af-ad9978d8997b-tls-certs\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.794786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.794779 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-dshm\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.795112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.795087 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-home\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.795205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.795097 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-tmp-dir\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.795257 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.795200 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-model-cache\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.797472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.797416 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-dshm\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.797593 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.797527 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38375563-5dc0-4101-93af-ad9978d8997b-tls-certs\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.857950 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.857910 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpqd\" (UniqueName: \"kubernetes.io/projected/38375563-5dc0-4101-93af-ad9978d8997b-kube-api-access-kqpqd\") pod \"distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:37.973805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:37.973711 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:38.392835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:38.392800 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh"] Nov 03 03:26:38.393192 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:26:38.393169 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38375563_5dc0_4101_93af_ad9978d8997b.slice/crio-d204316c2c6e4682f8af51d41eed7fc51d1210e9475b808b116b2a73ab1c266b WatchSource:0}: Error finding container d204316c2c6e4682f8af51d41eed7fc51d1210e9475b808b116b2a73ab1c266b: Status 404 returned error can't find the container with id d204316c2c6e4682f8af51d41eed7fc51d1210e9475b808b116b2a73ab1c266b Nov 03 03:26:39.039352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:39.039308 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" event={"ID":"38375563-5dc0-4101-93af-ad9978d8997b","Type":"ContainerStarted","Data":"db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b"} Nov 03 03:26:39.039352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:39.039353 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" event={"ID":"38375563-5dc0-4101-93af-ad9978d8997b","Type":"ContainerStarted","Data":"d204316c2c6e4682f8af51d41eed7fc51d1210e9475b808b116b2a73ab1c266b"} Nov 03 03:26:39.159621 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:39.159558 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" podStartSLOduration=2.15954342 podStartE2EDuration="2.15954342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:39.157234195 +0000 UTC m=+1534.187832734" watchObservedRunningTime="2026-11-03 03:26:39.15954342 +0000 UTC m=+1534.190141958" Nov 03 03:26:39.974152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:39.974120 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:45.589471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:45.578537 5117 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.828428 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:45.828380 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:45.895663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:45.895566 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-7cb55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:26:45.943470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:45.914527 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.490521 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:46.486754 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:26:46.490521 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:46.487657 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 03:26:47.974940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:47.974899 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:47.987785 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:47.987755 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:26:48.220496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:26:48.220435 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 03:27:00.011830 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:27:00.011793 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:27:18.271993 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.271955 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt"] Nov 03 03:27:18.276203 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.276186 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.318222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.318196 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-metrics-certs\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.318361 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.318237 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvtz\" (UniqueName: \"kubernetes.io/projected/4704b889-b624-4053-941c-3847608e5e8e-kube-api-access-fcvtz\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.318407 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.318355 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.318536 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.318420 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4704b889-b624-4053-941c-3847608e5e8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.381424 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.381389 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d82a79-serving-cert\"" Nov 03 03:27:18.384717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.384688 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt"] Nov 03 03:27:18.420097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.420058 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.420292 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.420168 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4704b889-b624-4053-941c-3847608e5e8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.420374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.420322 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-metrics-certs\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.420374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.420361 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvtz\" (UniqueName: \"kubernetes.io/projected/4704b889-b624-4053-941c-3847608e5e8e-kube-api-access-fcvtz\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.420870 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:27:18.420848 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-serving-cert: secret "maas-api-e2e-api-a-d82a79-serving-cert" not found Nov 03 03:27:18.420966 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:27:18.420930 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls podName:4704b889-b624-4053-941c-3847608e5e8e nodeName:}" failed. No retries permitted until 2026-11-03 03:27:18.920908966 +0000 UTC m=+1573.951507483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls") pod "maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" (UID: "4704b889-b624-4053-941c-3847608e5e8e") : secret "maas-api-e2e-api-a-d82a79-serving-cert" not found Nov 03 03:27:18.421697 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.421665 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4704b889-b624-4053-941c-3847608e5e8e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.424834 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.424804 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-metrics-certs\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.482620 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.482588 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvtz\" (UniqueName: \"kubernetes.io/projected/4704b889-b624-4053-941c-3847608e5e8e-kube-api-access-fcvtz\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.929265 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.929229 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:18.931848 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:18.931819 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls\") pod \"maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:19.190481 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:19.190353 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:19.571175 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:19.571140 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt"] Nov 03 03:27:19.572072 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:27:19.572041 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4704b889_b624_4053_941c_3847608e5e8e.slice/crio-5e7547d1adf6efa9996fd35a09fcce5b16e30ed8abe56807737e562488eb5ea2 WatchSource:0}: Error finding container 5e7547d1adf6efa9996fd35a09fcce5b16e30ed8abe56807737e562488eb5ea2: Status 404 returned error can't find the container with id 5e7547d1adf6efa9996fd35a09fcce5b16e30ed8abe56807737e562488eb5ea2 Nov 03 03:27:20.418541 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:20.418500 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" event={"ID":"4704b889-b624-4053-941c-3847608e5e8e","Type":"ContainerStarted","Data":"5e7547d1adf6efa9996fd35a09fcce5b16e30ed8abe56807737e562488eb5ea2"} Nov 03 03:27:21.427526 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:21.427464 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" event={"ID":"4704b889-b624-4053-941c-3847608e5e8e","Type":"ContainerStarted","Data":"24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3"} Nov 03 03:27:21.428038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:21.427791 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:21.547607 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:21.547546 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" podStartSLOduration=2.576751805 podStartE2EDuration="3.547530412s" totalImagesPullingTime="970.778607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:21.547370019 +0000 UTC m=+1576.577968592" watchObservedRunningTime="2026-11-03 03:27:21.547530412 +0000 UTC m=+1576.578128950" Nov 03 03:27:27.439697 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:27.439666 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:27:30.011553 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:27:30.011519 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:27:33.152431 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.152399 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs"] Nov 03 03:27:33.156530 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.156513 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.261258 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.261228 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9902a0c2\"/\"openshift-service-ca.crt\"" Nov 03 03:27:33.261498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.261283 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9902a0c2\"/\"default-dockercfg-6lpfz\"" Nov 03 03:27:33.261498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.261434 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9902a0c2\"/\"kube-root-ca.crt\"" Nov 03 03:27:33.261498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.261478 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9902a0c2\"/\"e2e-lifecycle-model-9902a0c2-kserve-self-signed-certs\"" Nov 03 03:27:33.264263 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.264230 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs"] Nov 03 03:27:33.308259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.308229 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-home\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.308437 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.308369 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-model-cache\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.308523 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.308506 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6a2d9-868b-4049-ad63-254b83d5a797-tls-certs\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.308589 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.308572 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn92q\" (UniqueName: \"kubernetes.io/projected/99d6a2d9-868b-4049-ad63-254b83d5a797-kube-api-access-bn92q\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.308640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.308626 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-tmp-dir\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.308684 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.308653 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-dshm\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.409625 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.409548 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-model-cache\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.409773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.409643 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6a2d9-868b-4049-ad63-254b83d5a797-tls-certs\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.409773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.409693 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn92q\" (UniqueName: \"kubernetes.io/projected/99d6a2d9-868b-4049-ad63-254b83d5a797-kube-api-access-bn92q\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.409773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.409736 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-tmp-dir\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.409773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.409752 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-dshm\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.410020 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.409799 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-home\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.410020 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.409951 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-model-cache\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.410164 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.410142 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-tmp-dir\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.410238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.410222 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-home\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.412186 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.412151 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-dshm\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.412372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.412355 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6a2d9-868b-4049-ad63-254b83d5a797-tls-certs\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.474590 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.474560 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn92q\" (UniqueName: \"kubernetes.io/projected/99d6a2d9-868b-4049-ad63-254b83d5a797-kube-api-access-bn92q\") pod \"e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:33.769384 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:33.769279 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:34.152240 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:34.152211 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs"] Nov 03 03:27:34.152654 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:27:34.152629 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99d6a2d9_868b_4049_ad63_254b83d5a797.slice/crio-4addd3cd693d7d5a02dd3543882d211168f41234c886ae4c5fd7bf6d9a9339b6 WatchSource:0}: Error finding container 4addd3cd693d7d5a02dd3543882d211168f41234c886ae4c5fd7bf6d9a9339b6: Status 404 returned error can't find the container with id 4addd3cd693d7d5a02dd3543882d211168f41234c886ae4c5fd7bf6d9a9339b6 Nov 03 03:27:34.511051 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:34.510949 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" event={"ID":"99d6a2d9-868b-4049-ad63-254b83d5a797","Type":"ContainerStarted","Data":"7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598"} Nov 03 03:27:34.511051 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:34.510997 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" event={"ID":"99d6a2d9-868b-4049-ad63-254b83d5a797","Type":"ContainerStarted","Data":"4addd3cd693d7d5a02dd3543882d211168f41234c886ae4c5fd7bf6d9a9339b6"} Nov 03 03:27:34.632953 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:34.632898 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" podStartSLOduration=1.632884784 podStartE2EDuration="1.632884784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:34.63036893 +0000 UTC m=+1589.660967468" watchObservedRunningTime="2026-11-03 03:27:34.632884784 +0000 UTC m=+1589.663483377" Nov 03 03:27:34.770514 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:34.770382 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:43.770143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:43.770103 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:43.782831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:43.782800 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:44.886314 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:44.886269 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:27:52.578609 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.578575 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt"] Nov 03 03:27:52.582587 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.582564 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.635865 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.635840 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b562f5-serving-cert\"" Nov 03 03:27:52.673826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.673794 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-metrics-certs\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.674017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.673853 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.674017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.673974 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f253604-523a-4f04-92b3-d4e4d3b02db8-openshift-service-ca\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.674110 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.674018 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4zn\" (UniqueName: \"kubernetes.io/projected/8f253604-523a-4f04-92b3-d4e4d3b02db8-kube-api-access-mw4zn\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.695247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.695211 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt"] Nov 03 03:27:52.775119 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.775089 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-metrics-certs\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.775278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.775133 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.775278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.775184 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f253604-523a-4f04-92b3-d4e4d3b02db8-openshift-service-ca\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.775278 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.775213 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4zn\" (UniqueName: \"kubernetes.io/projected/8f253604-523a-4f04-92b3-d4e4d3b02db8-kube-api-access-mw4zn\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.775381 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:27:52.775282 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-serving-cert: secret "maas-api-e2e-api-a-b562f5-serving-cert" not found Nov 03 03:27:52.775381 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:27:52.775336 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls podName:8f253604-523a-4f04-92b3-d4e4d3b02db8 nodeName:}" failed. No retries permitted until 2026-11-03 03:27:53.275321429 +0000 UTC m=+1608.305919950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls") pod "maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" (UID: "8f253604-523a-4f04-92b3-d4e4d3b02db8") : secret "maas-api-e2e-api-a-b562f5-serving-cert" not found Nov 03 03:27:52.775938 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.775917 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f253604-523a-4f04-92b3-d4e4d3b02db8-openshift-service-ca\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.777853 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.777832 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-metrics-certs\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:52.839205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:52.839127 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4zn\" (UniqueName: \"kubernetes.io/projected/8f253604-523a-4f04-92b3-d4e4d3b02db8-kube-api-access-mw4zn\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:53.282272 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:53.282194 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:53.284802 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:53.284774 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls\") pod \"maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:53.497237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:53.497207 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:53.877143 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:27:53.877113 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f253604_523a_4f04_92b3_d4e4d3b02db8.slice/crio-6ad17512cf604b3df65ec2bff97ff327d31c94c8841ca110adb64987fbb9982a WatchSource:0}: Error finding container 6ad17512cf604b3df65ec2bff97ff327d31c94c8841ca110adb64987fbb9982a: Status 404 returned error can't find the container with id 6ad17512cf604b3df65ec2bff97ff327d31c94c8841ca110adb64987fbb9982a Nov 03 03:27:53.877514 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:53.877311 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt"] Nov 03 03:27:54.663414 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:54.663373 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" event={"ID":"8f253604-523a-4f04-92b3-d4e4d3b02db8","Type":"ContainerStarted","Data":"6ad17512cf604b3df65ec2bff97ff327d31c94c8841ca110adb64987fbb9982a"} Nov 03 03:27:55.671540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:55.671493 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" event={"ID":"8f253604-523a-4f04-92b3-d4e4d3b02db8","Type":"ContainerStarted","Data":"58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923"} Nov 03 03:27:55.671983 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:55.671692 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:27:55.799245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:27:55.799195 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" podStartSLOduration=2.97864751 podStartE2EDuration="3.799180585s" totalImagesPullingTime="820.533075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:55.796700473 +0000 UTC m=+1610.827299009" watchObservedRunningTime="2026-11-03 03:27:55.799180585 +0000 UTC m=+1610.829779123" Nov 03 03:28:00.011937 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:28:00.011901 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:28:01.683700 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:01.683665 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:28:15.243978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.243947 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p"] Nov 03 03:28:15.249133 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.249112 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.320993 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.320944 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p"] Nov 03 03:28:15.322637 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.322610 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-home\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.322779 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.322737 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpzm\" (UniqueName: \"kubernetes.io/projected/6735ef43-0f84-4832-b8be-6947a0401721-kube-api-access-8lpzm\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.322894 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.322879 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-model-cache\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.322962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.322902 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6735ef43-0f84-4832-b8be-6947a0401721-tls-certs\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.322962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.322958 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-tmp-dir\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.323070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.322996 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-dshm\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.354820 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.354763 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"distinct-2-w0-0aa6b9-kserve-self-signed-certs\"" Nov 03 03:28:15.424671 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.424628 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpzm\" (UniqueName: \"kubernetes.io/projected/6735ef43-0f84-4832-b8be-6947a0401721-kube-api-access-8lpzm\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.424878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.424812 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-model-cache\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.424878 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.424846 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6735ef43-0f84-4832-b8be-6947a0401721-tls-certs\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.424995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.424900 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-tmp-dir\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.424995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.424971 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-dshm\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.425108 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.425001 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-home\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.425574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.425533 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-home\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.426428 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.426397 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-tmp-dir\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.426755 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.426713 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-model-cache\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.428431 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.428413 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6735ef43-0f84-4832-b8be-6947a0401721-tls-certs\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.429037 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.429016 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-dshm\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.496101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.496012 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpzm\" (UniqueName: \"kubernetes.io/projected/6735ef43-0f84-4832-b8be-6947a0401721-kube-api-access-8lpzm\") pod \"distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.561045 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.560999 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:15.968850 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:15.968821 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p"] Nov 03 03:28:15.970064 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:28:15.970037 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6735ef43_0f84_4832_b8be_6947a0401721.slice/crio-e30aeb7cc0ad7d55de7f235f33cd1e9e05fc69a494e837f157b46aacecb54f90 WatchSource:0}: Error finding container e30aeb7cc0ad7d55de7f235f33cd1e9e05fc69a494e837f157b46aacecb54f90: Status 404 returned error can't find the container with id e30aeb7cc0ad7d55de7f235f33cd1e9e05fc69a494e837f157b46aacecb54f90 Nov 03 03:28:16.864354 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:16.864316 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" event={"ID":"6735ef43-0f84-4832-b8be-6947a0401721","Type":"ContainerStarted","Data":"b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736"} Nov 03 03:28:16.864788 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:16.864363 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" event={"ID":"6735ef43-0f84-4832-b8be-6947a0401721","Type":"ContainerStarted","Data":"e30aeb7cc0ad7d55de7f235f33cd1e9e05fc69a494e837f157b46aacecb54f90"} Nov 03 03:28:16.986922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:16.986872 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" podStartSLOduration=1.986859028 podStartE2EDuration="1.986859028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:16.983134598 +0000 UTC m=+1632.013733136" watchObservedRunningTime="2026-11-03 03:28:16.986859028 +0000 UTC m=+1632.017457566" Nov 03 03:28:17.561222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:17.561190 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:25.561519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:25.561476 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:25.574850 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:25.574822 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:25.934847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:28:25.934766 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:28:30.011889 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:28:30.011852 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:29:00.011215 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:00.011169 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:29:03.093671 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.093630 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87"] Nov 03 03:29:03.097953 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.097935 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.152687 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.152651 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4da1f5-serving-cert\"" Nov 03 03:29:03.180582 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.180547 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.180760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.180619 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvzl\" (UniqueName: \"kubernetes.io/projected/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-kube-api-access-8bvzl\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.180760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.180710 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-metrics-certs\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.180760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.180748 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.208716 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.208687 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87"] Nov 03 03:29:03.281868 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.281838 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.282062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.281911 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvzl\" (UniqueName: \"kubernetes.io/projected/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-kube-api-access-8bvzl\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.282062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.281946 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-metrics-certs\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.282062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.281979 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.282380 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:03.282352 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-serving-cert: secret "maas-api-e2e-api-b-4da1f5-serving-cert" not found Nov 03 03:29:03.282560 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:03.282435 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls podName:3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7 nodeName:}" failed. No retries permitted until 2026-11-03 03:29:03.782416237 +0000 UTC m=+1678.813014767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls") pod "maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" (UID: "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7") : secret "maas-api-e2e-api-b-4da1f5-serving-cert" not found Nov 03 03:29:03.282807 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.282787 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.284790 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.284767 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-metrics-certs\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.344842 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.344771 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvzl\" (UniqueName: \"kubernetes.io/projected/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-kube-api-access-8bvzl\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.788946 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.788859 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:03.791663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:03.791635 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls\") pod \"maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:04.013432 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:04.013393 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:04.393628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:04.393594 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87"] Nov 03 03:29:04.394383 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:29:04.394360 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3df31a_1e3d_402a_bc18_a19fcfbb3bb7.slice/crio-ca4f1c8b0dd07e74f8f6f87ee880aac10722d1ae40b0e734e850b113a803ddd1 WatchSource:0}: Error finding container ca4f1c8b0dd07e74f8f6f87ee880aac10722d1ae40b0e734e850b113a803ddd1: Status 404 returned error can't find the container with id ca4f1c8b0dd07e74f8f6f87ee880aac10722d1ae40b0e734e850b113a803ddd1 Nov 03 03:29:05.220344 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:05.220226 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" event={"ID":"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7","Type":"ContainerStarted","Data":"ca4f1c8b0dd07e74f8f6f87ee880aac10722d1ae40b0e734e850b113a803ddd1"} Nov 03 03:29:06.228214 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:06.228162 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" event={"ID":"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7","Type":"ContainerStarted","Data":"c4373a1876b5bcce42c8c037b61b832396f74b5e0ae603d9b49ec75fa21c5032"} Nov 03 03:29:06.228761 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:06.228504 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:06.350327 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:06.350276 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" podStartSLOduration=2.4286218489999998 podStartE2EDuration="3.350262084s" totalImagesPullingTime="921.640235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:06.348573309 +0000 UTC m=+1681.379171848" watchObservedRunningTime="2026-11-03 03:29:06.350262084 +0000 UTC m=+1681.380860622" Nov 03 03:29:07.886114 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:07.886067 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg"] Nov 03 03:29:07.890811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:07.890786 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:07.944241 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:07.944206 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0aa6b9-openshift-default-dockercfg-57xw2\"" Nov 03 03:29:07.999842 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:07.999797 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg"] Nov 03 03:29:08.041792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.041752 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.041988 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.041800 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884b5\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-kube-api-access-884b5\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.042048 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.041980 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.042143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.042120 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.042207 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.042157 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.042207 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.042191 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.042313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.042224 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.042369 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.042312 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.042413 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.042393 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e875d1-b7e9-45f2-8d30-498be58ac927-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144151 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144038 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144151 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144103 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884b5\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-kube-api-access-884b5\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144222 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144293 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144319 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144346 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144374 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144711 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144427 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144711 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144515 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e875d1-b7e9-45f2-8d30-498be58ac927-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144711 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144528 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144711 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144680 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144704 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.144995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.144964 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.145251 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.145227 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e875d1-b7e9-45f2-8d30-498be58ac927-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.146941 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.146918 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.147100 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.147078 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.206580 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.206549 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.206745 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.206669 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-884b5\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-kube-api-access-884b5\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.505396 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.505294 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:08.941488 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.941420 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg"] Nov 03 03:29:08.941855 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:29:08.941692 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e875d1_b7e9_45f2_8d30_498be58ac927.slice/crio-8cc6573e89ffdce6863d18fafd79bdaadfdd85e1ab8a10025dd1dccade8944b6 WatchSource:0}: Error finding container 8cc6573e89ffdce6863d18fafd79bdaadfdd85e1ab8a10025dd1dccade8944b6: Status 404 returned error can't find the container with id 8cc6573e89ffdce6863d18fafd79bdaadfdd85e1ab8a10025dd1dccade8944b6 Nov 03 03:29:08.944807 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.944658 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:29:08.944807 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.944741 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:29:08.944807 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:08.944784 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:29:09.253468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:09.253365 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" event={"ID":"a8e875d1-b7e9-45f2-8d30-498be58ac927","Type":"ContainerStarted","Data":"90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741"} Nov 03 03:29:09.253468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:09.253418 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" event={"ID":"a8e875d1-b7e9-45f2-8d30-498be58ac927","Type":"ContainerStarted","Data":"8cc6573e89ffdce6863d18fafd79bdaadfdd85e1ab8a10025dd1dccade8944b6"} Nov 03 03:29:09.380689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:09.380638 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podStartSLOduration=2.380624037 podStartE2EDuration="2.380624037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:09.376557293 +0000 UTC m=+1684.407155833" watchObservedRunningTime="2026-11-03 03:29:09.380624037 +0000 UTC m=+1684.411222590" Nov 03 03:29:09.507895 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:09.507543 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:10.524780 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:10.524729 5117 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:29:10.525258 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:10.524808 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 03:29:11.518925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:11.518883 5117 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:29:11.519233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:11.519202 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 03:29:12.242208 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:12.242171 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:29:12.530803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:12.530711 5117 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:29:12.531107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:12.531074 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 03:29:13.524472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:13.520963 5117 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:29:13.524472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:13.521061 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 03:29:14.521959 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:14.521916 5117 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:29:14.522544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:14.522509 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 03:29:15.505912 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:15.505878 5117 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 03:29:15.506661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:15.506630 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 03:29:16.593956 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:16.593916 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:16.594534 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:16.594364 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:16.595341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:16.595320 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:29:20.652677 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.652640 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7116-74c99e"] Nov 03 03:29:20.656983 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.656962 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:20.713031 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.712992 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-74c99e"] Nov 03 03:29:20.746935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.746898 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7s9l\" (UniqueName: \"kubernetes.io/projected/822417eb-c1b4-42f5-90b5-b851671e3d59-kube-api-access-r7s9l\") pod \"test-curl-7116-74c99e\" (UID: \"822417eb-c1b4-42f5-90b5-b851671e3d59\") " pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:20.759401 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.759371 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr9t9\"" Nov 03 03:29:20.848099 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.848059 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s9l\" (UniqueName: \"kubernetes.io/projected/822417eb-c1b4-42f5-90b5-b851671e3d59-kube-api-access-r7s9l\") pod \"test-curl-7116-74c99e\" (UID: \"822417eb-c1b4-42f5-90b5-b851671e3d59\") " pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:20.910636 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.910550 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s9l\" (UniqueName: \"kubernetes.io/projected/822417eb-c1b4-42f5-90b5-b851671e3d59-kube-api-access-r7s9l\") pod \"test-curl-7116-74c99e\" (UID: \"822417eb-c1b4-42f5-90b5-b851671e3d59\") " pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:20.970374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:20.970340 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:21.346840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:21.346811 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-74c99e"] Nov 03 03:29:21.347061 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:29:21.347038 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod822417eb_c1b4_42f5_90b5_b851671e3d59.slice/crio-d9ffa279410ddf0e62dafb0765fa3517e8c647edc12d140ad7ab9304a3f2faa8 WatchSource:0}: Error finding container d9ffa279410ddf0e62dafb0765fa3517e8c647edc12d140ad7ab9304a3f2faa8: Status 404 returned error can't find the container with id d9ffa279410ddf0e62dafb0765fa3517e8c647edc12d140ad7ab9304a3f2faa8 Nov 03 03:29:22.346950 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:22.346904 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-74c99e" event={"ID":"822417eb-c1b4-42f5-90b5-b851671e3d59","Type":"ContainerStarted","Data":"d9ffa279410ddf0e62dafb0765fa3517e8c647edc12d140ad7ab9304a3f2faa8"} Nov 03 03:29:23.361597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:23.361555 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-74c99e" event={"ID":"822417eb-c1b4-42f5-90b5-b851671e3d59","Type":"ContainerStarted","Data":"c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02"} Nov 03 03:29:23.480375 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:23.480319 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7116-74c99e" podStartSLOduration=2.444405161 podStartE2EDuration="3.480305957s" totalImagesPullingTime="1.035900796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:23.478117934 +0000 UTC m=+1698.508716473" watchObservedRunningTime="2026-11-03 03:29:23.480305957 +0000 UTC m=+1698.510904495" Nov 03 03:29:25.061198 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:25.061165 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7116-74c99e"] Nov 03 03:29:25.068116 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:25.068087 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7116-74c99e"] Nov 03 03:29:25.375182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:25.375077 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7116-74c99e" podUID="822417eb-c1b4-42f5-90b5-b851671e3d59" containerName="test-curl-7116-74c99e" containerID="cri-o://c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02" gracePeriod=2 Nov 03 03:29:27.669856 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:27.669826 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:27.753738 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:27.753702 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822417eb-c1b4-42f5-90b5-b851671e3d59-kube-api-access-r7s9l\" (UniqueName: \"kubernetes.io/projected/822417eb-c1b4-42f5-90b5-b851671e3d59-kube-api-access-r7s9l\") pod \"822417eb-c1b4-42f5-90b5-b851671e3d59\" (UID: \"822417eb-c1b4-42f5-90b5-b851671e3d59\") " Nov 03 03:29:27.756032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:27.756004 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822417eb-c1b4-42f5-90b5-b851671e3d59-kube-api-access-r7s9l" pod "822417eb-c1b4-42f5-90b5-b851671e3d59" (UID: "822417eb-c1b4-42f5-90b5-b851671e3d59"). InnerVolumeSpecName "kube-api-access-r7s9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:29:27.855960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:27.855924 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7s9l\" (UniqueName: \"kubernetes.io/projected/822417eb-c1b4-42f5-90b5-b851671e3d59-kube-api-access-r7s9l\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:29:28.400003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:28.399970 5117 generic.go:356] "Generic (PLEG): container finished" podID="822417eb-c1b4-42f5-90b5-b851671e3d59" containerID="c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02" exitCode=137 Nov 03 03:29:28.400190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:28.400011 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:28.400190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:28.400064 5117 scope.go:122] "RemoveContainer" containerID="c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02" Nov 03 03:29:28.411497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:28.411476 5117 scope.go:122] "RemoveContainer" containerID="c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02" Nov 03 03:29:28.411782 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:28.411759 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02\": container with ID starting with c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02 not found: ID does not exist" containerID="c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02" Nov 03 03:29:28.411856 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:28.411791 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02"} err="failed to get container status \"c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02\": rpc error: code = NotFound desc = could not find container \"c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02\": container with ID starting with c9b6aab5d84dac25fdc16d171f9b87dfaba4437c3c324d112b049ab127888c02 not found: ID does not exist" Nov 03 03:29:28.461614 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:28.461580 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-74c99e\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="822417eb-c1b4-42f5-90b5-b851671e3d59" pod="opendatahub/test-curl-7116-74c99e" Nov 03 03:29:29.090679 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:29.090646 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822417eb-c1b4-42f5-90b5-b851671e3d59" path="/var/lib/kubelet/pods/822417eb-c1b4-42f5-90b5-b851671e3d59/volumes" Nov 03 03:29:30.011738 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:30.011693 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:29:43.360508 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.360470 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6"] Nov 03 03:29:43.361092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.361079 5117 cpu_manager.go:414] "containerMap: removing container" podUID="822417eb-c1b4-42f5-90b5-b851671e3d59" containerName="test-curl-7116-74c99e" Nov 03 03:29:43.361143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.361093 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822417eb-c1b4-42f5-90b5-b851671e3d59" containerName="test-curl-7116-74c99e" Nov 03 03:29:43.361186 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.361177 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="822417eb-c1b4-42f5-90b5-b851671e3d59" containerName="test-curl-7116-74c99e" Nov 03 03:29:43.366043 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.366025 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.423210 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.423173 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6"] Nov 03 03:29:43.469766 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.469728 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-398727-serving-cert\"" Nov 03 03:29:43.476983 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.476948 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98htz\" (UniqueName: \"kubernetes.io/projected/4fe4e32f-20fe-4181-9040-1a877e5e04ce-kube-api-access-98htz\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.477139 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.477119 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-metrics-certs\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.477258 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.477241 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4e32f-20fe-4181-9040-1a877e5e04ce-openshift-service-ca\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.477328 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.477275 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.578629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.578590 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98htz\" (UniqueName: \"kubernetes.io/projected/4fe4e32f-20fe-4181-9040-1a877e5e04ce-kube-api-access-98htz\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.578826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.578721 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-metrics-certs\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.578898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.578832 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4e32f-20fe-4181-9040-1a877e5e04ce-openshift-service-ca\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.578898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.578864 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.579084 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:43.579059 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-398727-serving-cert: secret "maas-api-e2e-api-b-398727-serving-cert" not found Nov 03 03:29:43.579220 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:29:43.579132 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls podName:4fe4e32f-20fe-4181-9040-1a877e5e04ce nodeName:}" failed. No retries permitted until 2026-11-03 03:29:44.079108954 +0000 UTC m=+1719.109707489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls") pod "maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" (UID: "4fe4e32f-20fe-4181-9040-1a877e5e04ce") : secret "maas-api-e2e-api-b-398727-serving-cert" not found Nov 03 03:29:43.579995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.579969 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4e32f-20fe-4181-9040-1a877e5e04ce-openshift-service-ca\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.582914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.582887 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-metrics-certs\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:43.679204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:43.679126 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98htz\" (UniqueName: \"kubernetes.io/projected/4fe4e32f-20fe-4181-9040-1a877e5e04ce-kube-api-access-98htz\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:44.086822 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:44.086792 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:44.089593 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:44.089563 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls\") pod \"maas-api-e2e-api-b-398727-7cfd9994cf-78tx6\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:44.280062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:44.280026 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:44.663336 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:44.663299 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6"] Nov 03 03:29:44.663790 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:29:44.663347 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe4e32f_20fe_4181_9040_1a877e5e04ce.slice/crio-5429f9f253beb220ccc62d98eedeb50a38d2fc54beca5648cbc86ebc8e6cc0c4 WatchSource:0}: Error finding container 5429f9f253beb220ccc62d98eedeb50a38d2fc54beca5648cbc86ebc8e6cc0c4: Status 404 returned error can't find the container with id 5429f9f253beb220ccc62d98eedeb50a38d2fc54beca5648cbc86ebc8e6cc0c4 Nov 03 03:29:45.525527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:45.525481 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" event={"ID":"4fe4e32f-20fe-4181-9040-1a877e5e04ce","Type":"ContainerStarted","Data":"5429f9f253beb220ccc62d98eedeb50a38d2fc54beca5648cbc86ebc8e6cc0c4"} Nov 03 03:29:46.534617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:46.534583 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" event={"ID":"4fe4e32f-20fe-4181-9040-1a877e5e04ce","Type":"ContainerStarted","Data":"5403f32b0e155a6c4a8936466fdb2c2bc445167be8aa3e318ea0a38534934890"} Nov 03 03:29:46.535149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:46.534793 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:29:46.654492 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:46.654406 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" podStartSLOduration=2.842918472 podStartE2EDuration="3.654393637s" totalImagesPullingTime="811.475165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:46.654364066 +0000 UTC m=+1721.684962604" watchObservedRunningTime="2026-11-03 03:29:46.654393637 +0000 UTC m=+1721.684992176" Nov 03 03:29:52.547723 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:29:52.547695 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:30:00.012760 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:30:00.012716 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:30:08.383963 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.383869 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb"] Nov 03 03:30:08.388393 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.388366 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.492621 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.492581 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-64c59d-serving-cert\"" Nov 03 03:30:08.496460 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.496408 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb"] Nov 03 03:30:08.531358 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.531310 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-metrics-certs\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.531580 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.531548 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd06f6a-072b-44da-946d-0e590525b929-openshift-service-ca\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.531674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.531582 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7ck\" (UniqueName: \"kubernetes.io/projected/ecd06f6a-072b-44da-946d-0e590525b929-kube-api-access-jn7ck\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.531674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.531609 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.633702 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.633650 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd06f6a-072b-44da-946d-0e590525b929-openshift-service-ca\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.633918 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.633723 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7ck\" (UniqueName: \"kubernetes.io/projected/ecd06f6a-072b-44da-946d-0e590525b929-kube-api-access-jn7ck\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.633918 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.633768 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.635596 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.633880 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-metrics-certs\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.637547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.637512 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd06f6a-072b-44da-946d-0e590525b929-openshift-service-ca\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.637889 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:30:08.637858 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-serving-cert: secret "maas-api-e2e-api-a-64c59d-serving-cert" not found Nov 03 03:30:08.638015 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:30:08.637938 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls podName:ecd06f6a-072b-44da-946d-0e590525b929 nodeName:}" failed. No retries permitted until 2026-11-03 03:30:09.137916384 +0000 UTC m=+1744.168514900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls") pod "maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" (UID: "ecd06f6a-072b-44da-946d-0e590525b929") : secret "maas-api-e2e-api-a-64c59d-serving-cert" not found Nov 03 03:30:08.639798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.639775 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-metrics-certs\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:08.701893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:08.701864 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7ck\" (UniqueName: \"kubernetes.io/projected/ecd06f6a-072b-44da-946d-0e590525b929-kube-api-access-jn7ck\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:09.152624 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:09.152588 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:09.155126 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:09.155103 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls\") pod \"maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:09.302414 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:09.302382 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:09.687570 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:30:09.687534 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd06f6a_072b_44da_946d_0e590525b929.slice/crio-49d0eee09c43891c227cfcfd739098a04d1c43bb7c3c8c2b8f61f33e1b5515a4 WatchSource:0}: Error finding container 49d0eee09c43891c227cfcfd739098a04d1c43bb7c3c8c2b8f61f33e1b5515a4: Status 404 returned error can't find the container with id 49d0eee09c43891c227cfcfd739098a04d1c43bb7c3c8c2b8f61f33e1b5515a4 Nov 03 03:30:09.688331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:09.688302 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb"] Nov 03 03:30:09.703879 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:09.703852 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" event={"ID":"ecd06f6a-072b-44da-946d-0e590525b929","Type":"ContainerStarted","Data":"49d0eee09c43891c227cfcfd739098a04d1c43bb7c3c8c2b8f61f33e1b5515a4"} Nov 03 03:30:10.712106 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:10.712069 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" event={"ID":"ecd06f6a-072b-44da-946d-0e590525b929","Type":"ContainerStarted","Data":"da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f"} Nov 03 03:30:10.712538 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:10.712137 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:10.849365 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:10.849309 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" podStartSLOduration=2.021405323 podStartE2EDuration="2.84929618s" totalImagesPullingTime="827.890857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:10.848577264 +0000 UTC m=+1745.879175802" watchObservedRunningTime="2026-11-03 03:30:10.84929618 +0000 UTC m=+1745.879894719" Nov 03 03:30:16.724526 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:16.724487 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:30:23.674105 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.674073 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt"] Nov 03 03:30:23.678103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.678087 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.775826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.775784 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnbh\" (UniqueName: \"kubernetes.io/projected/c781c61b-dc25-4a42-ac30-7838bb9254b8-kube-api-access-rpnbh\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.775992 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.775966 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.776049 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.776014 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-metrics-certs\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.776247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.776218 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c781c61b-dc25-4a42-ac30-7838bb9254b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.783667 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.783645 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-372186-serving-cert\"" Nov 03 03:30:23.787252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.787230 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt"] Nov 03 03:30:23.877919 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.877887 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.878101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.877953 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-metrics-certs\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.878101 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:30:23.878026 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-372186-serving-cert: secret "maas-api-e2e-api-a-372186-serving-cert" not found Nov 03 03:30:23.878234 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:30:23.878106 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls podName:c781c61b-dc25-4a42-ac30-7838bb9254b8 nodeName:}" failed. No retries permitted until 2026-11-03 03:30:24.378090293 +0000 UTC m=+1759.408688828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls") pod "maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" (UID: "c781c61b-dc25-4a42-ac30-7838bb9254b8") : secret "maas-api-e2e-api-a-372186-serving-cert" not found Nov 03 03:30:23.878234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.878025 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c781c61b-dc25-4a42-ac30-7838bb9254b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.878234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.878155 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnbh\" (UniqueName: \"kubernetes.io/projected/c781c61b-dc25-4a42-ac30-7838bb9254b8-kube-api-access-rpnbh\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.878729 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.878707 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c781c61b-dc25-4a42-ac30-7838bb9254b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.880780 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.880755 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-metrics-certs\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:23.941221 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:23.941149 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnbh\" (UniqueName: \"kubernetes.io/projected/c781c61b-dc25-4a42-ac30-7838bb9254b8-kube-api-access-rpnbh\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:24.384872 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:24.384838 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:24.387488 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:24.387432 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls\") pod \"maas-api-e2e-api-a-372186-64b84b58ff-hs8jt\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:24.592314 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:24.592277 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:24.981140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:24.981108 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt"] Nov 03 03:30:24.981655 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:30:24.981633 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc781c61b_dc25_4a42_ac30_7838bb9254b8.slice/crio-676b03146e6d0e91b1ac439adcc5de6382b3f2a0974367d5e6045e9b3651aaa6 WatchSource:0}: Error finding container 676b03146e6d0e91b1ac439adcc5de6382b3f2a0974367d5e6045e9b3651aaa6: Status 404 returned error can't find the container with id 676b03146e6d0e91b1ac439adcc5de6382b3f2a0974367d5e6045e9b3651aaa6 Nov 03 03:30:25.836857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:25.836817 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" event={"ID":"c781c61b-dc25-4a42-ac30-7838bb9254b8","Type":"ContainerStarted","Data":"676b03146e6d0e91b1ac439adcc5de6382b3f2a0974367d5e6045e9b3651aaa6"} Nov 03 03:30:26.857883 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:26.857835 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" event={"ID":"c781c61b-dc25-4a42-ac30-7838bb9254b8","Type":"ContainerStarted","Data":"d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331"} Nov 03 03:30:26.858351 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:26.857923 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:27.032733 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:27.029154 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" podStartSLOduration=3.202528771 podStartE2EDuration="4.029136927s" totalImagesPullingTime="826.608156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:27.02753998 +0000 UTC m=+1762.058138518" watchObservedRunningTime="2026-11-03 03:30:27.029136927 +0000 UTC m=+1762.059735466" Nov 03 03:30:30.012215 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:30:30.012170 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:30:32.870169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:32.870134 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:30:37.980003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:37.979963 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt"] Nov 03 03:30:37.980563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:37.980177 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" podUID="4704b889-b624-4053-941c-3847608e5e8e" containerName="maas-api" containerID="cri-o://24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3" gracePeriod=30 Nov 03 03:30:38.314624 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.314575 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:30:38.386810 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.386772 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls\") pod \"4704b889-b624-4053-941c-3847608e5e8e\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " Nov 03 03:30:38.387004 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.386927 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4704b889-b624-4053-941c-3847608e5e8e-kube-api-access-fcvtz\" (UniqueName: \"kubernetes.io/projected/4704b889-b624-4053-941c-3847608e5e8e-kube-api-access-fcvtz\") pod \"4704b889-b624-4053-941c-3847608e5e8e\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " Nov 03 03:30:38.387004 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.386966 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-metrics-certs\") pod \"4704b889-b624-4053-941c-3847608e5e8e\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " Nov 03 03:30:38.387124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.387045 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4704b889-b624-4053-941c-3847608e5e8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4704b889-b624-4053-941c-3847608e5e8e-openshift-service-ca\") pod \"4704b889-b624-4053-941c-3847608e5e8e\" (UID: \"4704b889-b624-4053-941c-3847608e5e8e\") " Nov 03 03:30:38.387946 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.387905 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4704b889-b624-4053-941c-3847608e5e8e-openshift-service-ca" pod "4704b889-b624-4053-941c-3847608e5e8e" (UID: "4704b889-b624-4053-941c-3847608e5e8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:38.389504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.389463 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls" pod "4704b889-b624-4053-941c-3847608e5e8e" (UID: "4704b889-b624-4053-941c-3847608e5e8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:30:38.389504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.389434 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4704b889-b624-4053-941c-3847608e5e8e-kube-api-access-fcvtz" pod "4704b889-b624-4053-941c-3847608e5e8e" (UID: "4704b889-b624-4053-941c-3847608e5e8e"). InnerVolumeSpecName "kube-api-access-fcvtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:38.389712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.389695 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-metrics-certs" pod "4704b889-b624-4053-941c-3847608e5e8e" (UID: "4704b889-b624-4053-941c-3847608e5e8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:30:38.489326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.489253 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:38.489326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.489295 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvtz\" (UniqueName: \"kubernetes.io/projected/4704b889-b624-4053-941c-3847608e5e8e-kube-api-access-fcvtz\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:38.489326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.489311 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4704b889-b624-4053-941c-3847608e5e8e-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:38.489326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.489325 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4704b889-b624-4053-941c-3847608e5e8e-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:38.943405 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.943353 5117 generic.go:356] "Generic (PLEG): container finished" podID="4704b889-b624-4053-941c-3847608e5e8e" containerID="24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3" exitCode=0 Nov 03 03:30:38.943656 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.943591 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" Nov 03 03:30:38.943656 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.943645 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" event={"ID":"4704b889-b624-4053-941c-3847608e5e8e","Type":"ContainerDied","Data":"24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3"} Nov 03 03:30:38.943784 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.943699 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt" event={"ID":"4704b889-b624-4053-941c-3847608e5e8e","Type":"ContainerDied","Data":"5e7547d1adf6efa9996fd35a09fcce5b16e30ed8abe56807737e562488eb5ea2"} Nov 03 03:30:38.943784 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.943730 5117 scope.go:122] "RemoveContainer" containerID="24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3" Nov 03 03:30:38.962335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.962311 5117 scope.go:122] "RemoveContainer" containerID="24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3" Nov 03 03:30:38.962729 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:30:38.962705 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3\": container with ID starting with 24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3 not found: ID does not exist" containerID="24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3" Nov 03 03:30:38.962952 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:38.962740 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3"} err="failed to get container status \"24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3\": rpc error: code = NotFound desc = could not find container \"24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3\": container with ID starting with 24c0d2aac23a8d2614ed7e59263fb59d88b1f3955b2b511ac18a54a86410b1b3 not found: ID does not exist" Nov 03 03:30:39.114788 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:39.114755 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt"] Nov 03 03:30:39.167288 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:39.167253 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d82a79-65fbfc6947-w8qqt"] Nov 03 03:30:41.097499 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:41.096627 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4704b889-b624-4053-941c-3847608e5e8e" path="/var/lib/kubelet/pods/4704b889-b624-4053-941c-3847608e5e8e/volumes" Nov 03 03:30:50.630409 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.630372 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm"] Nov 03 03:30:50.631009 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.630995 5117 cpu_manager.go:414] "containerMap: removing container" podUID="4704b889-b624-4053-941c-3847608e5e8e" containerName="maas-api" Nov 03 03:30:50.631054 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.631010 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4704b889-b624-4053-941c-3847608e5e8e" containerName="maas-api" Nov 03 03:30:50.631142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.631133 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="4704b889-b624-4053-941c-3847608e5e8e" containerName="maas-api" Nov 03 03:30:50.634595 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.634578 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687069 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687033 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687069 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687073 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687282 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687162 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687282 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687239 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687354 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687304 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kc92\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-kube-api-access-9kc92\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687392 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687351 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687392 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687374 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687485 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687545 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687528 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.687735 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.687720 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-398727-openshift-default-dockercfg-ckr6k\"" Nov 03 03:30:50.788966 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.788919 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.788970 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kc92\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-kube-api-access-9kc92\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789020 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789050 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789358 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789186 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789358 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789250 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789492 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789430 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789492 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789474 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789529 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789540 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789778 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789749 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789811 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.789981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.789913 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.790021 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.790001 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.791892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.791870 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.791985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.791967 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.846394 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.846357 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm"] Nov 03 03:30:50.852849 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.852816 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kc92\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-kube-api-access-9kc92\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.858053 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.858002 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:50.948230 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:50.948133 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:51.383958 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:51.383927 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm"] Nov 03 03:30:51.384084 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:30:51.384065 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2172e4f8_2783_4e80_aa86_ad3dbd82f750.slice/crio-f0e219e648090def90c58dc07d59d6ad73125584726d3c282c9ecb11ad2defac WatchSource:0}: Error finding container f0e219e648090def90c58dc07d59d6ad73125584726d3c282c9ecb11ad2defac: Status 404 returned error can't find the container with id f0e219e648090def90c58dc07d59d6ad73125584726d3c282c9ecb11ad2defac Nov 03 03:30:51.386638 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:51.386598 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:30:51.386720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:51.386673 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:30:51.386720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:51.386706 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:30:52.042163 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:52.042118 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" event={"ID":"2172e4f8-2783-4e80-aa86-ad3dbd82f750","Type":"ContainerStarted","Data":"30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d"} Nov 03 03:30:52.042606 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:52.042190 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" event={"ID":"2172e4f8-2783-4e80-aa86-ad3dbd82f750","Type":"ContainerStarted","Data":"f0e219e648090def90c58dc07d59d6ad73125584726d3c282c9ecb11ad2defac"} Nov 03 03:30:52.271112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:52.271054 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podStartSLOduration=2.271040432 podStartE2EDuration="2.271040432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:52.217411134 +0000 UTC m=+1787.248009673" watchObservedRunningTime="2026-11-03 03:30:52.271040432 +0000 UTC m=+1787.301638970" Nov 03 03:30:52.948738 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:52.948694 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:52.950029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:52.950001 5117 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 03:30:52.950184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:52.950058 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 03:30:53.948836 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:53.948801 5117 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 03:30:53.949233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:53.948851 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 03:30:54.746461 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:54.746395 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87"] Nov 03 03:30:54.747584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:54.747548 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" podUID="3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" containerName="maas-api" containerID="cri-o://c4373a1876b5bcce42c8c037b61b832396f74b5e0ae603d9b49ec75fa21c5032" gracePeriod=30 Nov 03 03:30:54.949280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:54.949193 5117 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 03:30:54.949280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:54.949255 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 03:30:55.119728 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.118109 5117 generic.go:356] "Generic (PLEG): container finished" podID="3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" containerID="c4373a1876b5bcce42c8c037b61b832396f74b5e0ae603d9b49ec75fa21c5032" exitCode=0 Nov 03 03:30:55.165469 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.159615 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" event={"ID":"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7","Type":"ContainerDied","Data":"c4373a1876b5bcce42c8c037b61b832396f74b5e0ae603d9b49ec75fa21c5032"} Nov 03 03:30:55.185847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.185817 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:30:55.285273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.285227 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-kube-api-access-8bvzl\" (UniqueName: \"kubernetes.io/projected/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-kube-api-access-8bvzl\") pod \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " Nov 03 03:30:55.285529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.285510 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls\") pod \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " Nov 03 03:30:55.285614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.285552 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-openshift-service-ca\") pod \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " Nov 03 03:30:55.285614 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.285598 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-metrics-certs\") pod \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\" (UID: \"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7\") " Nov 03 03:30:55.285975 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.285933 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-openshift-service-ca" pod "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" (UID: "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:30:55.286221 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.286189 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:55.288035 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.287988 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-metrics-certs" pod "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" (UID: "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:30:55.289959 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.289749 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-kube-api-access-8bvzl" pod "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" (UID: "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7"). InnerVolumeSpecName "kube-api-access-8bvzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:30:55.289959 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.289917 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls" pod "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" (UID: "3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:30:55.388874 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.388776 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:55.388874 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.388816 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:55.388874 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.388833 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvzl\" (UniqueName: \"kubernetes.io/projected/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7-kube-api-access-8bvzl\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:30:55.957465 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.955105 5117 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 03:30:55.957465 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:55.955172 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 03:30:56.140487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:56.140387 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" event={"ID":"3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7","Type":"ContainerDied","Data":"ca4f1c8b0dd07e74f8f6f87ee880aac10722d1ae40b0e734e850b113a803ddd1"} Nov 03 03:30:56.140685 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:56.140524 5117 scope.go:122] "RemoveContainer" containerID="c4373a1876b5bcce42c8c037b61b832396f74b5e0ae603d9b49ec75fa21c5032" Nov 03 03:30:56.140762 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:56.140746 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87" Nov 03 03:30:56.331065 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:56.331027 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87"] Nov 03 03:30:56.335024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:56.334998 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4da1f5-5bdb5d6846-hft87"] Nov 03 03:30:56.948940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:56.948905 5117 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 03:30:56.949134 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:56.948994 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 03:30:57.091351 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:57.091321 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" path="/var/lib/kubelet/pods/3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7/volumes" Nov 03 03:30:57.953852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:57.953820 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:57.954233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:57.954215 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:30:57.954938 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:30:57.954917 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:31:00.018656 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:31:00.018616 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:31:06.768387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:06.768341 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn"] Nov 03 03:31:06.769202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:06.769148 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" podUID="59763795-5ea6-428f-b90a-165ddb42ba90" containerName="maas-api" containerID="cri-o://d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32" gracePeriod=30 Nov 03 03:31:07.191522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.191491 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:31:07.236147 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.236113 5117 generic.go:356] "Generic (PLEG): container finished" podID="59763795-5ea6-428f-b90a-165ddb42ba90" containerID="d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32" exitCode=0 Nov 03 03:31:07.236420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.236201 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" event={"ID":"59763795-5ea6-428f-b90a-165ddb42ba90","Type":"ContainerDied","Data":"d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32"} Nov 03 03:31:07.236420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.236292 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" event={"ID":"59763795-5ea6-428f-b90a-165ddb42ba90","Type":"ContainerDied","Data":"434f09a741840968783019951c59aac7f7ab8e2b1a792b957605725c05b54e11"} Nov 03 03:31:07.236420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.236345 5117 scope.go:122] "RemoveContainer" containerID="d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32" Nov 03 03:31:07.237619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.236190 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn" Nov 03 03:31:07.255906 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.255320 5117 scope.go:122] "RemoveContainer" containerID="d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32" Nov 03 03:31:07.255906 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:31:07.255886 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32\": container with ID starting with d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32 not found: ID does not exist" containerID="d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32" Nov 03 03:31:07.256030 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.255924 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32"} err="failed to get container status \"d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32\": rpc error: code = NotFound desc = could not find container \"d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32\": container with ID starting with d397f75836b381fe4ee2edde1f9b6050c5bd781eb7099f919bd6f441c2824d32 not found: ID does not exist" Nov 03 03:31:07.282056 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.282013 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59763795-5ea6-428f-b90a-165ddb42ba90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59763795-5ea6-428f-b90a-165ddb42ba90-openshift-service-ca\") pod \"59763795-5ea6-428f-b90a-165ddb42ba90\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " Nov 03 03:31:07.282238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.282184 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls\") pod \"59763795-5ea6-428f-b90a-165ddb42ba90\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " Nov 03 03:31:07.282380 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.282242 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59763795-5ea6-428f-b90a-165ddb42ba90-kube-api-access-mcdwl\" (UniqueName: \"kubernetes.io/projected/59763795-5ea6-428f-b90a-165ddb42ba90-kube-api-access-mcdwl\") pod \"59763795-5ea6-428f-b90a-165ddb42ba90\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " Nov 03 03:31:07.282436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.282414 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59763795-5ea6-428f-b90a-165ddb42ba90-openshift-service-ca" pod "59763795-5ea6-428f-b90a-165ddb42ba90" (UID: "59763795-5ea6-428f-b90a-165ddb42ba90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:07.283787 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.282709 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-metrics-certs\") pod \"59763795-5ea6-428f-b90a-165ddb42ba90\" (UID: \"59763795-5ea6-428f-b90a-165ddb42ba90\") " Nov 03 03:31:07.283787 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.283417 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59763795-5ea6-428f-b90a-165ddb42ba90-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:07.285042 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.285016 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls" pod "59763795-5ea6-428f-b90a-165ddb42ba90" (UID: "59763795-5ea6-428f-b90a-165ddb42ba90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:07.285385 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.285358 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59763795-5ea6-428f-b90a-165ddb42ba90-kube-api-access-mcdwl" pod "59763795-5ea6-428f-b90a-165ddb42ba90" (UID: "59763795-5ea6-428f-b90a-165ddb42ba90"). InnerVolumeSpecName "kube-api-access-mcdwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:07.286560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.286529 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-metrics-certs" pod "59763795-5ea6-428f-b90a-165ddb42ba90" (UID: "59763795-5ea6-428f-b90a-165ddb42ba90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:07.384520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.384489 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:07.384520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.384521 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdwl\" (UniqueName: \"kubernetes.io/projected/59763795-5ea6-428f-b90a-165ddb42ba90-kube-api-access-mcdwl\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:07.384782 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.384537 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59763795-5ea6-428f-b90a-165ddb42ba90-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:07.385697 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.385672 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:31:07.387552 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.387533 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:31:07.664065 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.664034 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn"] Nov 03 03:31:07.667347 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.667320 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-f5qvn"] Nov 03 03:31:07.959234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.959189 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn"] Nov 03 03:31:07.959781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:07.959464 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" containerID="cri-o://4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e" gracePeriod=30 Nov 03 03:31:09.091720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:09.091687 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59763795-5ea6-428f-b90a-165ddb42ba90" path="/var/lib/kubelet/pods/59763795-5ea6-428f-b90a-165ddb42ba90/volumes" Nov 03 03:31:13.273081 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.273045 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:31:13.278936 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.278900 5117 generic.go:356] "Generic (PLEG): container finished" podID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerID="4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e" exitCode=0 Nov 03 03:31:13.279097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.278951 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" Nov 03 03:31:13.279097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.278982 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" event={"ID":"65f18784-21f2-4fa4-b018-4bf4a5de259f","Type":"ContainerDied","Data":"4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e"} Nov 03 03:31:13.279097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.279051 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn" event={"ID":"65f18784-21f2-4fa4-b018-4bf4a5de259f","Type":"ContainerDied","Data":"c3f68b1230df44de0bb93a62685fdce499badd5439a86963fb76c8b0113de0cf"} Nov 03 03:31:13.279097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.279082 5117 scope.go:122] "RemoveContainer" containerID="4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e" Nov 03 03:31:13.290130 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.290105 5117 scope.go:122] "RemoveContainer" containerID="4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e" Nov 03 03:31:13.290468 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:31:13.290426 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e\": container with ID starting with 4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e not found: ID does not exist" containerID="4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e" Nov 03 03:31:13.290535 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.290481 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e"} err="failed to get container status \"4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e\": rpc error: code = NotFound desc = could not find container \"4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e\": container with ID starting with 4e5870a455c919b36d7ea58739d312efa49cc4b1e5e12fb480d59aeb0dbf0e8e not found: ID does not exist" Nov 03 03:31:13.369382 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369351 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-data\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.369634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369425 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-token\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-token\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.369634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369517 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-kube-api-access-w5fhp\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-kube-api-access-w5fhp\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.369634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369588 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f18784-21f2-4fa4-b018-4bf4a5de259f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f18784-21f2-4fa4-b018-4bf4a5de259f-istiod-ca-cert\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.369806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369715 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-certs\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.369806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369745 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-envoy\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.369806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369765 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-data" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.369806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369780 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-podinfo\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.370022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369883 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-socket\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.370022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.369941 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-credential-socket\") pod \"65f18784-21f2-4fa4-b018-4bf4a5de259f\" (UID: \"65f18784-21f2-4fa4-b018-4bf4a5de259f\") " Nov 03 03:31:13.370123 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370070 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f18784-21f2-4fa4-b018-4bf4a5de259f-istiod-ca-cert" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:13.370375 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370321 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-certs" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.370375 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370339 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-socket" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.370693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370605 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-credential-socket" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.370756 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370746 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.370814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370761 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.370814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370770 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.370814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370778 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.370814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.370786 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f18784-21f2-4fa4-b018-4bf4a5de259f-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.372281 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.372256 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-token" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:13.372281 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.372256 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-envoy" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:13.372410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.372294 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-kube-api-access-w5fhp" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "kube-api-access-w5fhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:13.372754 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.372735 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-podinfo" pod "65f18784-21f2-4fa4-b018-4bf4a5de259f" (UID: "65f18784-21f2-4fa4-b018-4bf4a5de259f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:13.471824 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.471785 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.471824 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.471821 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.471824 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.471834 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.472089 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.471847 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5fhp\" (UniqueName: \"kubernetes.io/projected/65f18784-21f2-4fa4-b018-4bf4a5de259f-kube-api-access-w5fhp\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:13.734390 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.734351 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn"] Nov 03 03:31:13.738423 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:13.738388 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-z57jn"] Nov 03 03:31:15.091175 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:15.091140 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" path="/var/lib/kubelet/pods/65f18784-21f2-4fa4-b018-4bf4a5de259f/volumes" Nov 03 03:31:23.803467 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:23.803078 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt"] Nov 03 03:31:23.803467 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:23.803334 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" podUID="8f253604-523a-4f04-92b3-d4e4d3b02db8" containerName="maas-api" containerID="cri-o://58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923" gracePeriod=30 Nov 03 03:31:24.130272 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.130213 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:31:24.145602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.145565 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls\") pod \"8f253604-523a-4f04-92b3-d4e4d3b02db8\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " Nov 03 03:31:24.145819 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.145754 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f253604-523a-4f04-92b3-d4e4d3b02db8-kube-api-access-mw4zn\" (UniqueName: \"kubernetes.io/projected/8f253604-523a-4f04-92b3-d4e4d3b02db8-kube-api-access-mw4zn\") pod \"8f253604-523a-4f04-92b3-d4e4d3b02db8\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " Nov 03 03:31:24.145893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.145828 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-metrics-certs\") pod \"8f253604-523a-4f04-92b3-d4e4d3b02db8\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " Nov 03 03:31:24.145893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.145872 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f253604-523a-4f04-92b3-d4e4d3b02db8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f253604-523a-4f04-92b3-d4e4d3b02db8-openshift-service-ca\") pod \"8f253604-523a-4f04-92b3-d4e4d3b02db8\" (UID: \"8f253604-523a-4f04-92b3-d4e4d3b02db8\") " Nov 03 03:31:24.146722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.146690 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f253604-523a-4f04-92b3-d4e4d3b02db8-openshift-service-ca" pod "8f253604-523a-4f04-92b3-d4e4d3b02db8" (UID: "8f253604-523a-4f04-92b3-d4e4d3b02db8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:24.148912 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.148885 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls" pod "8f253604-523a-4f04-92b3-d4e4d3b02db8" (UID: "8f253604-523a-4f04-92b3-d4e4d3b02db8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:24.164962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.164915 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-metrics-certs" pod "8f253604-523a-4f04-92b3-d4e4d3b02db8" (UID: "8f253604-523a-4f04-92b3-d4e4d3b02db8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:31:24.165117 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.164990 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f253604-523a-4f04-92b3-d4e4d3b02db8-kube-api-access-mw4zn" pod "8f253604-523a-4f04-92b3-d4e4d3b02db8" (UID: "8f253604-523a-4f04-92b3-d4e4d3b02db8"). InnerVolumeSpecName "kube-api-access-mw4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:24.247467 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.247406 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw4zn\" (UniqueName: \"kubernetes.io/projected/8f253604-523a-4f04-92b3-d4e4d3b02db8-kube-api-access-mw4zn\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:24.247467 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.247469 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:24.247727 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.247488 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f253604-523a-4f04-92b3-d4e4d3b02db8-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:24.247727 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.247503 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f253604-523a-4f04-92b3-d4e4d3b02db8-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:24.382540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.381860 5117 generic.go:356] "Generic (PLEG): container finished" podID="8f253604-523a-4f04-92b3-d4e4d3b02db8" containerID="58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923" exitCode=0 Nov 03 03:31:24.382540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.382147 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" event={"ID":"8f253604-523a-4f04-92b3-d4e4d3b02db8","Type":"ContainerDied","Data":"58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923"} Nov 03 03:31:24.382540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.382210 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" event={"ID":"8f253604-523a-4f04-92b3-d4e4d3b02db8","Type":"ContainerDied","Data":"6ad17512cf604b3df65ec2bff97ff327d31c94c8841ca110adb64987fbb9982a"} Nov 03 03:31:24.382540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.382237 5117 scope.go:122] "RemoveContainer" containerID="58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923" Nov 03 03:31:24.382540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.382367 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt" Nov 03 03:31:24.399377 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.399349 5117 scope.go:122] "RemoveContainer" containerID="58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923" Nov 03 03:31:24.399993 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:31:24.399918 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923\": container with ID starting with 58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923 not found: ID does not exist" containerID="58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923" Nov 03 03:31:24.399993 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.399958 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923"} err="failed to get container status \"58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923\": rpc error: code = NotFound desc = could not find container \"58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923\": container with ID starting with 58178ab6cbe73cbfd450b635e2ff67d28ddfb2a48326e25f540161ca92a4a923 not found: ID does not exist" Nov 03 03:31:24.554539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.554469 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt"] Nov 03 03:31:24.559483 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:24.559424 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b562f5-5bfc6f96cd-lbckt"] Nov 03 03:31:25.094700 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:25.094664 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f253604-523a-4f04-92b3-d4e4d3b02db8" path="/var/lib/kubelet/pods/8f253604-523a-4f04-92b3-d4e4d3b02db8/volumes" Nov 03 03:31:30.013259 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:31:30.013220 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:31:37.224765 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:37.224729 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8"] Nov 03 03:31:37.225249 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:37.224935 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" containerID="cri-o://47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9" gracePeriod=30 Nov 03 03:31:42.493581 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.493499 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:31:42.515720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515694 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-envoy\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.515880 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515760 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-certs\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.515880 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515784 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-kube-api-access-x67ql\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-kube-api-access-x67ql\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.515880 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515829 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79913da2-9bfb-45da-8c50-7e081c821789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79913da2-9bfb-45da-8c50-7e081c821789-istio-podinfo\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.515880 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515864 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-credential-socket\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.516107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515901 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-socket\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.516107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515920 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-data\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.516107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.515975 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-istio-token\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-istio-token\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.516107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516057 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79913da2-9bfb-45da-8c50-7e081c821789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79913da2-9bfb-45da-8c50-7e081c821789-istiod-ca-cert\") pod \"79913da2-9bfb-45da-8c50-7e081c821789\" (UID: \"79913da2-9bfb-45da-8c50-7e081c821789\") " Nov 03 03:31:42.516107 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516073 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-certs" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:42.516378 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516140 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-credential-socket" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:42.516378 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516160 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-socket" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:42.516378 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516297 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-data" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:42.516746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516657 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.516746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516676 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.516746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516684 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.516746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516693 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.516746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.516731 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79913da2-9bfb-45da-8c50-7e081c821789-istiod-ca-cert" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:31:42.518165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.518142 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-envoy" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:31:42.518165 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.518147 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-kube-api-access-x67ql" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "kube-api-access-x67ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:42.518356 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.518298 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-istio-token" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:31:42.518566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.518535 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79913da2-9bfb-45da-8c50-7e081c821789-istio-podinfo" pod "79913da2-9bfb-45da-8c50-7e081c821789" (UID: "79913da2-9bfb-45da-8c50-7e081c821789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:31:42.541697 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.541671 5117 generic.go:356] "Generic (PLEG): container finished" podID="79913da2-9bfb-45da-8c50-7e081c821789" containerID="47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9" exitCode=0 Nov 03 03:31:42.541846 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.541827 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" Nov 03 03:31:42.541898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.541851 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" event={"ID":"79913da2-9bfb-45da-8c50-7e081c821789","Type":"ContainerDied","Data":"47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9"} Nov 03 03:31:42.541939 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.541899 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8" event={"ID":"79913da2-9bfb-45da-8c50-7e081c821789","Type":"ContainerDied","Data":"ff560f2460d25e6e07e3a36de54904fb6c51ebfbd06ae8155b66478f91256eaf"} Nov 03 03:31:42.541939 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.541921 5117 scope.go:122] "RemoveContainer" containerID="47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9" Nov 03 03:31:42.552823 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.552806 5117 scope.go:122] "RemoveContainer" containerID="47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9" Nov 03 03:31:42.553095 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:31:42.553076 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9\": container with ID starting with 47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9 not found: ID does not exist" containerID="47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9" Nov 03 03:31:42.553156 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.553106 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9"} err="failed to get container status \"47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9\": rpc error: code = NotFound desc = could not find container \"47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9\": container with ID starting with 47b41128cb1b13df99ac59a089ea6413eed688d96df04ca3a4ea1ee9ede2a7c9 not found: ID does not exist" Nov 03 03:31:42.617934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.617902 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67ql\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-kube-api-access-x67ql\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.617934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.617931 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79913da2-9bfb-45da-8c50-7e081c821789-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.617934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.617942 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79913da2-9bfb-45da-8c50-7e081c821789-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.618176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.617953 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79913da2-9bfb-45da-8c50-7e081c821789-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.618176 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.617962 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79913da2-9bfb-45da-8c50-7e081c821789-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:31:42.783270 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.783184 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8"] Nov 03 03:31:42.787122 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:42.787092 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-v4kw8"] Nov 03 03:31:43.090725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:31:43.090688 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79913da2-9bfb-45da-8c50-7e081c821789" path="/var/lib/kubelet/pods/79913da2-9bfb-45da-8c50-7e081c821789/volumes" Nov 03 03:32:00.052286 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:32:00.052243 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:32:27.551939 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.551908 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6"] Nov 03 03:32:27.552435 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.552107 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" podUID="4fe4e32f-20fe-4181-9040-1a877e5e04ce" containerName="maas-api" containerID="cri-o://5403f32b0e155a6c4a8936466fdb2c2bc445167be8aa3e318ea0a38534934890" gracePeriod=30 Nov 03 03:32:27.907220 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.907191 5117 generic.go:356] "Generic (PLEG): container finished" podID="4fe4e32f-20fe-4181-9040-1a877e5e04ce" containerID="5403f32b0e155a6c4a8936466fdb2c2bc445167be8aa3e318ea0a38534934890" exitCode=0 Nov 03 03:32:27.907349 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.907272 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" event={"ID":"4fe4e32f-20fe-4181-9040-1a877e5e04ce","Type":"ContainerDied","Data":"5403f32b0e155a6c4a8936466fdb2c2bc445167be8aa3e318ea0a38534934890"} Nov 03 03:32:27.907349 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.907328 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" event={"ID":"4fe4e32f-20fe-4181-9040-1a877e5e04ce","Type":"ContainerDied","Data":"5429f9f253beb220ccc62d98eedeb50a38d2fc54beca5648cbc86ebc8e6cc0c4"} Nov 03 03:32:27.907349 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.907345 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5429f9f253beb220ccc62d98eedeb50a38d2fc54beca5648cbc86ebc8e6cc0c4" Nov 03 03:32:27.937380 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.933963 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:32:27.971002 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.970965 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-metrics-certs\") pod \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " Nov 03 03:32:27.971192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.971017 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe4e32f-20fe-4181-9040-1a877e5e04ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4e32f-20fe-4181-9040-1a877e5e04ce-openshift-service-ca\") pod \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " Nov 03 03:32:27.971192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.971105 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe4e32f-20fe-4181-9040-1a877e5e04ce-kube-api-access-98htz\" (UniqueName: \"kubernetes.io/projected/4fe4e32f-20fe-4181-9040-1a877e5e04ce-kube-api-access-98htz\") pod \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " Nov 03 03:32:27.971192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.971138 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls\") pod \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\" (UID: \"4fe4e32f-20fe-4181-9040-1a877e5e04ce\") " Nov 03 03:32:27.971730 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.971671 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe4e32f-20fe-4181-9040-1a877e5e04ce-openshift-service-ca" pod "4fe4e32f-20fe-4181-9040-1a877e5e04ce" (UID: "4fe4e32f-20fe-4181-9040-1a877e5e04ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:32:27.971936 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.971915 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4e32f-20fe-4181-9040-1a877e5e04ce-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:27.975674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.975648 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-metrics-certs" pod "4fe4e32f-20fe-4181-9040-1a877e5e04ce" (UID: "4fe4e32f-20fe-4181-9040-1a877e5e04ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:32:27.976218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.976179 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe4e32f-20fe-4181-9040-1a877e5e04ce-kube-api-access-98htz" pod "4fe4e32f-20fe-4181-9040-1a877e5e04ce" (UID: "4fe4e32f-20fe-4181-9040-1a877e5e04ce"). InnerVolumeSpecName "kube-api-access-98htz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:27.976218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:27.976189 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls" pod "4fe4e32f-20fe-4181-9040-1a877e5e04ce" (UID: "4fe4e32f-20fe-4181-9040-1a877e5e04ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:32:28.074916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:28.074876 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:28.075131 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:28.075112 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98htz\" (UniqueName: \"kubernetes.io/projected/4fe4e32f-20fe-4181-9040-1a877e5e04ce-kube-api-access-98htz\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:28.075249 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:28.075234 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe4e32f-20fe-4181-9040-1a877e5e04ce-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:28.914686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:28.914653 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6" Nov 03 03:32:29.090702 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.090661 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6"] Nov 03 03:32:29.136239 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.136206 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-398727-7cfd9994cf-78tx6"] Nov 03 03:32:29.159962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.159929 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c"] Nov 03 03:32:29.160497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160483 5117 cpu_manager.go:414] "containerMap: removing container" podUID="4fe4e32f-20fe-4181-9040-1a877e5e04ce" containerName="maas-api" Nov 03 03:32:29.160548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160498 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe4e32f-20fe-4181-9040-1a877e5e04ce" containerName="maas-api" Nov 03 03:32:29.160548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160509 5117 cpu_manager.go:414] "containerMap: removing container" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" Nov 03 03:32:29.160548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160515 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" Nov 03 03:32:29.160548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160524 5117 cpu_manager.go:414] "containerMap: removing container" podUID="59763795-5ea6-428f-b90a-165ddb42ba90" containerName="maas-api" Nov 03 03:32:29.160548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160529 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59763795-5ea6-428f-b90a-165ddb42ba90" containerName="maas-api" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160553 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8f253604-523a-4f04-92b3-d4e4d3b02db8" containerName="maas-api" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160557 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f253604-523a-4f04-92b3-d4e4d3b02db8" containerName="maas-api" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160565 5117 cpu_manager.go:414] "containerMap: removing container" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160570 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160583 5117 cpu_manager.go:414] "containerMap: removing container" podUID="3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" containerName="maas-api" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160587 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" containerName="maas-api" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160678 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3df31a-1e3d-402a-bc18-a19fcfbb3bb7" containerName="maas-api" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160690 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe4e32f-20fe-4181-9040-1a877e5e04ce" containerName="maas-api" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160697 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="79913da2-9bfb-45da-8c50-7e081c821789" containerName="istio-proxy" Nov 03 03:32:29.160701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160703 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="59763795-5ea6-428f-b90a-165ddb42ba90" containerName="maas-api" Nov 03 03:32:29.160982 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160712 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f253604-523a-4f04-92b3-d4e4d3b02db8" containerName="maas-api" Nov 03 03:32:29.160982 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.160718 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f18784-21f2-4fa4-b018-4bf4a5de259f" containerName="istio-proxy" Nov 03 03:32:29.164155 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.164134 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.217562 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.217532 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-211ed3-serving-cert\"" Nov 03 03:32:29.225281 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.225253 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c"] Nov 03 03:32:29.290780 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.290748 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-metrics-certs\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.290933 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.290789 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6ncd\" (UniqueName: \"kubernetes.io/projected/b49ed2d8-458a-4464-ac2b-50788bd13f40-kube-api-access-l6ncd\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.290933 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.290910 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49ed2d8-458a-4464-ac2b-50788bd13f40-openshift-service-ca\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.291005 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.290944 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-maas-api-tls\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.391911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.391872 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49ed2d8-458a-4464-ac2b-50788bd13f40-openshift-service-ca\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.391911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.391911 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-maas-api-tls\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.392126 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.392037 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-metrics-certs\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.392126 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.392079 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ncd\" (UniqueName: \"kubernetes.io/projected/b49ed2d8-458a-4464-ac2b-50788bd13f40-kube-api-access-l6ncd\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.392538 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.392517 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49ed2d8-458a-4464-ac2b-50788bd13f40-openshift-service-ca\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.394567 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.394545 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-maas-api-tls\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.394651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.394583 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-metrics-certs\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.454401 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.454310 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ncd\" (UniqueName: \"kubernetes.io/projected/b49ed2d8-458a-4464-ac2b-50788bd13f40-kube-api-access-l6ncd\") pod \"maas-api-e2e-api-b-211ed3-87677f65f-t9j7c\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.477048 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.477022 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:29.874530 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.874501 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c"] Nov 03 03:32:29.875362 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:32:29.875336 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49ed2d8_458a_4464_ac2b_50788bd13f40.slice/crio-c57e1171d6358c9c6e188187297762513ad148bf16822c9cc29d4b59f8ac7c89 WatchSource:0}: Error finding container c57e1171d6358c9c6e188187297762513ad148bf16822c9cc29d4b59f8ac7c89: Status 404 returned error can't find the container with id c57e1171d6358c9c6e188187297762513ad148bf16822c9cc29d4b59f8ac7c89 Nov 03 03:32:29.876748 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.876730 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:32:29.922028 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:29.921991 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" event={"ID":"b49ed2d8-458a-4464-ac2b-50788bd13f40","Type":"ContainerStarted","Data":"c57e1171d6358c9c6e188187297762513ad148bf16822c9cc29d4b59f8ac7c89"} Nov 03 03:32:30.011757 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:32:30.011718 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:32:30.933190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:30.933150 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" event={"ID":"b49ed2d8-458a-4464-ac2b-50788bd13f40","Type":"ContainerStarted","Data":"afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c"} Nov 03 03:32:30.933617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:30.933353 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:31.055597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:31.055499 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" podStartSLOduration=1.192923276 podStartE2EDuration="2.055484592s" totalImagesPullingTime="862.561316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:31.053583189 +0000 UTC m=+1886.084181728" watchObservedRunningTime="2026-11-03 03:32:31.055484592 +0000 UTC m=+1886.086083129" Nov 03 03:32:31.090967 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:31.090922 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe4e32f-20fe-4181-9040-1a877e5e04ce" path="/var/lib/kubelet/pods/4fe4e32f-20fe-4181-9040-1a877e5e04ce/volumes" Nov 03 03:32:36.945297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:36.945268 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:32:41.031108 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:41.031074 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm"] Nov 03 03:32:41.031693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:41.031360 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" containerID="cri-o://30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d" gracePeriod=30 Nov 03 03:32:42.960827 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:42.960793 5117 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:32:42.961397 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:42.960847 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:32:46.313493 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.313467 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:32:46.418404 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418372 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-token\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-token\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418404 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418408 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-certs\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418662 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418462 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istiod-ca-cert\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418662 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418501 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-podinfo\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418662 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418648 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-envoy\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418825 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418703 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-certs" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.418825 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418720 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-kube-api-access-9kc92\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-kube-api-access-9kc92\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418825 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418797 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-data\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418834 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-socket\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.418984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418893 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-credential-socket\") pod \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\" (UID: \"2172e4f8-2783-4e80-aa86-ad3dbd82f750\") " Nov 03 03:32:46.419105 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.418978 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istiod-ca-cert" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:32:46.419329 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419262 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-socket" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.419613 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419480 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-credential-socket" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.419708 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419633 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-data" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.419998 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419902 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.419998 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419930 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.419998 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419946 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.419998 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419974 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.419998 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.419988 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.421001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.420963 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-podinfo" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:32:46.421001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.420964 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-token" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:46.421132 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.421021 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-envoy" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:32:46.421647 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.421628 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-kube-api-access-9kc92" pod "2172e4f8-2783-4e80-aa86-ad3dbd82f750" (UID: "2172e4f8-2783-4e80-aa86-ad3dbd82f750"). InnerVolumeSpecName "kube-api-access-9kc92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:32:46.520882 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.520840 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.520882 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.520868 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kc92\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-kube-api-access-9kc92\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.520882 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.520882 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:46.520882 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:46.520890 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2172e4f8-2783-4e80-aa86-ad3dbd82f750-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:32:47.051641 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.051605 5117 generic.go:356] "Generic (PLEG): container finished" podID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerID="30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d" exitCode=0 Nov 03 03:32:47.051830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.051659 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" Nov 03 03:32:47.051830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.051689 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" event={"ID":"2172e4f8-2783-4e80-aa86-ad3dbd82f750","Type":"ContainerDied","Data":"30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d"} Nov 03 03:32:47.051830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.051749 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm" event={"ID":"2172e4f8-2783-4e80-aa86-ad3dbd82f750","Type":"ContainerDied","Data":"f0e219e648090def90c58dc07d59d6ad73125584726d3c282c9ecb11ad2defac"} Nov 03 03:32:47.051830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.051771 5117 scope.go:122] "RemoveContainer" containerID="30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d" Nov 03 03:32:47.063181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.063163 5117 scope.go:122] "RemoveContainer" containerID="30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d" Nov 03 03:32:47.063470 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:32:47.063427 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d\": container with ID starting with 30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d not found: ID does not exist" containerID="30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d" Nov 03 03:32:47.063517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.063472 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d"} err="failed to get container status \"30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d\": rpc error: code = NotFound desc = could not find container \"30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d\": container with ID starting with 30c4624bf824e67ed397542482f7150de6e1dae14f638a2a51840a213604272d not found: ID does not exist" Nov 03 03:32:47.272309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.272273 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm"] Nov 03 03:32:47.323517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:47.323487 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-m2vnm"] Nov 03 03:32:49.090685 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:32:49.090652 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" path="/var/lib/kubelet/pods/2172e4f8-2783-4e80-aa86-ad3dbd82f750/volumes" Nov 03 03:33:00.076743 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:33:00.076705 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:33:07.107936 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.107900 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7116-98694c"] Nov 03 03:33:07.108501 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.108485 5117 cpu_manager.go:414] "containerMap: removing container" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" Nov 03 03:33:07.108557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.108503 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" Nov 03 03:33:07.108659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.108648 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="2172e4f8-2783-4e80-aa86-ad3dbd82f750" containerName="istio-proxy" Nov 03 03:33:07.113485 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.113439 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:07.165745 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.165708 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-98694c"] Nov 03 03:33:07.192218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.192182 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmpxq\" (UniqueName: \"kubernetes.io/projected/32636981-4bf4-46f0-8c56-54738881439a-kube-api-access-hmpxq\") pod \"test-curl-7116-98694c\" (UID: \"32636981-4bf4-46f0-8c56-54738881439a\") " pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:07.212095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.212065 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr9t9\"" Nov 03 03:33:07.293765 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.293675 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmpxq\" (UniqueName: \"kubernetes.io/projected/32636981-4bf4-46f0-8c56-54738881439a-kube-api-access-hmpxq\") pod \"test-curl-7116-98694c\" (UID: \"32636981-4bf4-46f0-8c56-54738881439a\") " pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:07.355733 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.355705 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmpxq\" (UniqueName: \"kubernetes.io/projected/32636981-4bf4-46f0-8c56-54738881439a-kube-api-access-hmpxq\") pod \"test-curl-7116-98694c\" (UID: \"32636981-4bf4-46f0-8c56-54738881439a\") " pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:07.425916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.425827 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:07.856235 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:33:07.856185 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32636981_4bf4_46f0_8c56_54738881439a.slice/crio-36e3e2c205a9cb1c992b520c9c347c1f07d3b363b29005168517fcd1c2f56f57 WatchSource:0}: Error finding container 36e3e2c205a9cb1c992b520c9c347c1f07d3b363b29005168517fcd1c2f56f57: Status 404 returned error can't find the container with id 36e3e2c205a9cb1c992b520c9c347c1f07d3b363b29005168517fcd1c2f56f57 Nov 03 03:33:07.856866 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:07.856832 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7116-98694c"] Nov 03 03:33:08.207913 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:08.207819 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-98694c" event={"ID":"32636981-4bf4-46f0-8c56-54738881439a","Type":"ContainerStarted","Data":"6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c"} Nov 03 03:33:08.207913 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:08.207868 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7116-98694c" event={"ID":"32636981-4bf4-46f0-8c56-54738881439a","Type":"ContainerStarted","Data":"36e3e2c205a9cb1c992b520c9c347c1f07d3b363b29005168517fcd1c2f56f57"} Nov 03 03:33:08.328939 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:08.328869 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7116-98694c" podStartSLOduration=1.328847305 podStartE2EDuration="1.328847305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:08.323582062 +0000 UTC m=+1923.354180600" watchObservedRunningTime="2026-11-03 03:33:08.328847305 +0000 UTC m=+1923.359445849" Nov 03 03:33:09.850589 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:09.850559 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7116-98694c"] Nov 03 03:33:09.856838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:09.856802 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7116-98694c"] Nov 03 03:33:10.225198 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:10.225099 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7116-98694c" podUID="32636981-4bf4-46f0-8c56-54738881439a" containerName="test-curl-7116-98694c" containerID="cri-o://6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c" gracePeriod=2 Nov 03 03:33:10.280211 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:33:10.280168 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7116-98694c\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="32636981-4bf4-46f0-8c56-54738881439a" pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:12.437794 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:12.437762 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:12.457293 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:12.457268 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32636981-4bf4-46f0-8c56-54738881439a-kube-api-access-hmpxq\" (UniqueName: \"kubernetes.io/projected/32636981-4bf4-46f0-8c56-54738881439a-kube-api-access-hmpxq\") pod \"32636981-4bf4-46f0-8c56-54738881439a\" (UID: \"32636981-4bf4-46f0-8c56-54738881439a\") " Nov 03 03:33:12.459560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:12.459529 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32636981-4bf4-46f0-8c56-54738881439a-kube-api-access-hmpxq" pod "32636981-4bf4-46f0-8c56-54738881439a" (UID: "32636981-4bf4-46f0-8c56-54738881439a"). InnerVolumeSpecName "kube-api-access-hmpxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:33:12.559610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:12.559588 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmpxq\" (UniqueName: \"kubernetes.io/projected/32636981-4bf4-46f0-8c56-54738881439a-kube-api-access-hmpxq\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:13.091198 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:13.091167 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32636981-4bf4-46f0-8c56-54738881439a" path="/var/lib/kubelet/pods/32636981-4bf4-46f0-8c56-54738881439a/volumes" Nov 03 03:33:13.247382 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:13.247349 5117 generic.go:356] "Generic (PLEG): container finished" podID="32636981-4bf4-46f0-8c56-54738881439a" containerID="6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c" exitCode=137 Nov 03 03:33:13.247577 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:13.247409 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7116-98694c" Nov 03 03:33:13.247577 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:13.247479 5117 scope.go:122] "RemoveContainer" containerID="6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c" Nov 03 03:33:13.257854 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:13.257833 5117 scope.go:122] "RemoveContainer" containerID="6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c" Nov 03 03:33:13.258187 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:33:13.258162 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c\": container with ID starting with 6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c not found: ID does not exist" containerID="6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c" Nov 03 03:33:13.258229 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:13.258200 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c"} err="failed to get container status \"6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c\": rpc error: code = NotFound desc = could not find container \"6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c\": container with ID starting with 6efc456d67e4536283aafab962c45dd33424c2ec29bb4d8b0b95944258d3f90c not found: ID does not exist" Nov 03 03:33:15.343623 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:15.343585 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs"] Nov 03 03:33:30.012305 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:33:30.012265 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:33:30.368690 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.368608 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" podUID="99d6a2d9-868b-4049-ad63-254b83d5a797" containerName="main" containerID="cri-o://7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598" gracePeriod=45 Nov 03 03:33:30.646512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.646485 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:33:30.804768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.804731 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-home\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-home\") pod \"99d6a2d9-868b-4049-ad63-254b83d5a797\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.804931 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-tmp-dir\") pod \"99d6a2d9-868b-4049-ad63-254b83d5a797\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805025 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d6a2d9-868b-4049-ad63-254b83d5a797-kube-api-access-bn92q\" (UniqueName: \"kubernetes.io/projected/99d6a2d9-868b-4049-ad63-254b83d5a797-kube-api-access-bn92q\") pod \"99d6a2d9-868b-4049-ad63-254b83d5a797\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805072 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-model-cache\") pod \"99d6a2d9-868b-4049-ad63-254b83d5a797\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805100 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-dshm\") pod \"99d6a2d9-868b-4049-ad63-254b83d5a797\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805158 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-home" pod "99d6a2d9-868b-4049-ad63-254b83d5a797" (UID: "99d6a2d9-868b-4049-ad63-254b83d5a797"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805178 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d6a2d9-868b-4049-ad63-254b83d5a797-tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6a2d9-868b-4049-ad63-254b83d5a797-tls-certs\") pod \"99d6a2d9-868b-4049-ad63-254b83d5a797\" (UID: \"99d6a2d9-868b-4049-ad63-254b83d5a797\") " Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805182 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-tmp-dir" pod "99d6a2d9-868b-4049-ad63-254b83d5a797" (UID: "99d6a2d9-868b-4049-ad63-254b83d5a797"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:30.805487 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805358 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-model-cache" pod "99d6a2d9-868b-4049-ad63-254b83d5a797" (UID: "99d6a2d9-868b-4049-ad63-254b83d5a797"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:30.806062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805820 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:30.806062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.805985 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:30.806062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.806010 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:30.807929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.807869 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-dshm" pod "99d6a2d9-868b-4049-ad63-254b83d5a797" (UID: "99d6a2d9-868b-4049-ad63-254b83d5a797"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:33:30.808507 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.808477 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d6a2d9-868b-4049-ad63-254b83d5a797-tls-certs" pod "99d6a2d9-868b-4049-ad63-254b83d5a797" (UID: "99d6a2d9-868b-4049-ad63-254b83d5a797"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:33:30.809788 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.809750 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d6a2d9-868b-4049-ad63-254b83d5a797-kube-api-access-bn92q" pod "99d6a2d9-868b-4049-ad63-254b83d5a797" (UID: "99d6a2d9-868b-4049-ad63-254b83d5a797"). InnerVolumeSpecName "kube-api-access-bn92q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:33:30.907335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.907259 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn92q\" (UniqueName: \"kubernetes.io/projected/99d6a2d9-868b-4049-ad63-254b83d5a797-kube-api-access-bn92q\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:30.907335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.907290 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6a2d9-868b-4049-ad63-254b83d5a797-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:30.907335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:30.907301 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6a2d9-868b-4049-ad63-254b83d5a797-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:31.417299 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.417262 5117 generic.go:356] "Generic (PLEG): container finished" podID="99d6a2d9-868b-4049-ad63-254b83d5a797" containerID="7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598" exitCode=0 Nov 03 03:33:31.417812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.417310 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" Nov 03 03:33:31.417812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.417344 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" event={"ID":"99d6a2d9-868b-4049-ad63-254b83d5a797","Type":"ContainerDied","Data":"7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598"} Nov 03 03:33:31.417812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.417390 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" event={"ID":"99d6a2d9-868b-4049-ad63-254b83d5a797","Type":"ContainerDied","Data":"4addd3cd693d7d5a02dd3543882d211168f41234c886ae4c5fd7bf6d9a9339b6"} Nov 03 03:33:31.417812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.417409 5117 scope.go:122] "RemoveContainer" containerID="7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598" Nov 03 03:33:31.428531 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.428498 5117 scope.go:122] "RemoveContainer" containerID="7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598" Nov 03 03:33:31.428925 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:33:31.428888 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598\": container with ID starting with 7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598 not found: ID does not exist" containerID="7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598" Nov 03 03:33:31.429032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.428939 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598"} err="failed to get container status \"7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598\": rpc error: code = NotFound desc = could not find container \"7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598\": container with ID starting with 7c09915189d3e213d1c3d866217ca259834ebb1fa6563a65c2f86669a5cfe598 not found: ID does not exist" Nov 03 03:33:31.586561 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.586520 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs"] Nov 03 03:33:31.597619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.597580 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs" podUID="99d6a2d9-868b-4049-ad63-254b83d5a797" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:33:31.637127 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:31.637081 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9902a0c2/e2e-lifecycle-model-9902a0c2-kserve-7fc44d95cc-5g7fs"] Nov 03 03:33:33.090862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:33.090825 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d6a2d9-868b-4049-ad63-254b83d5a797" path="/var/lib/kubelet/pods/99d6a2d9-868b-4049-ad63-254b83d5a797/volumes" Nov 03 03:33:57.697986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:57.697945 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb"] Nov 03 03:33:57.698571 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:57.698190 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" podUID="ecd06f6a-072b-44da-946d-0e590525b929" containerName="maas-api" containerID="cri-o://da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f" gracePeriod=30 Nov 03 03:33:58.040029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.039998 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:33:58.090652 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.090611 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls\") pod \"ecd06f6a-072b-44da-946d-0e590525b929\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " Nov 03 03:33:58.090838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.090662 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd06f6a-072b-44da-946d-0e590525b929-kube-api-access-jn7ck\" (UniqueName: \"kubernetes.io/projected/ecd06f6a-072b-44da-946d-0e590525b929-kube-api-access-jn7ck\") pod \"ecd06f6a-072b-44da-946d-0e590525b929\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " Nov 03 03:33:58.090838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.090824 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd06f6a-072b-44da-946d-0e590525b929-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd06f6a-072b-44da-946d-0e590525b929-openshift-service-ca\") pod \"ecd06f6a-072b-44da-946d-0e590525b929\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " Nov 03 03:33:58.090973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.090873 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-metrics-certs\") pod \"ecd06f6a-072b-44da-946d-0e590525b929\" (UID: \"ecd06f6a-072b-44da-946d-0e590525b929\") " Nov 03 03:33:58.091237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.091203 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd06f6a-072b-44da-946d-0e590525b929-openshift-service-ca" pod "ecd06f6a-072b-44da-946d-0e590525b929" (UID: "ecd06f6a-072b-44da-946d-0e590525b929"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:33:58.091689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.091666 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd06f6a-072b-44da-946d-0e590525b929-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:58.093137 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.093102 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls" pod "ecd06f6a-072b-44da-946d-0e590525b929" (UID: "ecd06f6a-072b-44da-946d-0e590525b929"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:33:58.093229 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.093115 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd06f6a-072b-44da-946d-0e590525b929-kube-api-access-jn7ck" pod "ecd06f6a-072b-44da-946d-0e590525b929" (UID: "ecd06f6a-072b-44da-946d-0e590525b929"). InnerVolumeSpecName "kube-api-access-jn7ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:33:58.093661 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.093640 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-metrics-certs" pod "ecd06f6a-072b-44da-946d-0e590525b929" (UID: "ecd06f6a-072b-44da-946d-0e590525b929"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:33:58.192723 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.192685 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:58.192723 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.192718 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd06f6a-072b-44da-946d-0e590525b929-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:58.192915 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.192733 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7ck\" (UniqueName: \"kubernetes.io/projected/ecd06f6a-072b-44da-946d-0e590525b929-kube-api-access-jn7ck\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:33:58.594175 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.594136 5117 generic.go:356] "Generic (PLEG): container finished" podID="ecd06f6a-072b-44da-946d-0e590525b929" containerID="da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f" exitCode=0 Nov 03 03:33:58.594353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.594186 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" Nov 03 03:33:58.594353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.594224 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" event={"ID":"ecd06f6a-072b-44da-946d-0e590525b929","Type":"ContainerDied","Data":"da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f"} Nov 03 03:33:58.594353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.594281 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb" event={"ID":"ecd06f6a-072b-44da-946d-0e590525b929","Type":"ContainerDied","Data":"49d0eee09c43891c227cfcfd739098a04d1c43bb7c3c8c2b8f61f33e1b5515a4"} Nov 03 03:33:58.594353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.594310 5117 scope.go:122] "RemoveContainer" containerID="da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f" Nov 03 03:33:58.605153 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.605139 5117 scope.go:122] "RemoveContainer" containerID="da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f" Nov 03 03:33:58.605420 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:33:58.605393 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f\": container with ID starting with da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f not found: ID does not exist" containerID="da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f" Nov 03 03:33:58.605544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.605429 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f"} err="failed to get container status \"da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f\": rpc error: code = NotFound desc = could not find container \"da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f\": container with ID starting with da2b10c4ea991195d01d5f3d14a68e95171e4f88e2a140d6bde73bff9faa272f not found: ID does not exist" Nov 03 03:33:58.768496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.768392 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb"] Nov 03 03:33:58.773499 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:58.773467 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-64c59d-6ffc79f645-lnzvb"] Nov 03 03:33:59.090704 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:33:59.090664 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd06f6a-072b-44da-946d-0e590525b929" path="/var/lib/kubelet/pods/ecd06f6a-072b-44da-946d-0e590525b929/volumes" Nov 03 03:34:00.011499 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:34:00.011430 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:34:16.823172 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:16.823134 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt"] Nov 03 03:34:16.823705 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:16.823415 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" podUID="c781c61b-dc25-4a42-ac30-7838bb9254b8" containerName="maas-api" containerID="cri-o://d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331" gracePeriod=30 Nov 03 03:34:17.166415 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.166387 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:34:17.281050 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.281016 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-metrics-certs\") pod \"c781c61b-dc25-4a42-ac30-7838bb9254b8\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " Nov 03 03:34:17.282271 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.281226 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c781c61b-dc25-4a42-ac30-7838bb9254b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c781c61b-dc25-4a42-ac30-7838bb9254b8-openshift-service-ca\") pod \"c781c61b-dc25-4a42-ac30-7838bb9254b8\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " Nov 03 03:34:17.282271 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.281260 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls\") pod \"c781c61b-dc25-4a42-ac30-7838bb9254b8\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " Nov 03 03:34:17.282271 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.281323 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c781c61b-dc25-4a42-ac30-7838bb9254b8-kube-api-access-rpnbh\" (UniqueName: \"kubernetes.io/projected/c781c61b-dc25-4a42-ac30-7838bb9254b8-kube-api-access-rpnbh\") pod \"c781c61b-dc25-4a42-ac30-7838bb9254b8\" (UID: \"c781c61b-dc25-4a42-ac30-7838bb9254b8\") " Nov 03 03:34:17.282746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.282385 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c781c61b-dc25-4a42-ac30-7838bb9254b8-openshift-service-ca" pod "c781c61b-dc25-4a42-ac30-7838bb9254b8" (UID: "c781c61b-dc25-4a42-ac30-7838bb9254b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:34:17.305272 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.305233 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls" pod "c781c61b-dc25-4a42-ac30-7838bb9254b8" (UID: "c781c61b-dc25-4a42-ac30-7838bb9254b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:34:17.328587 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.328059 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c781c61b-dc25-4a42-ac30-7838bb9254b8-kube-api-access-rpnbh" pod "c781c61b-dc25-4a42-ac30-7838bb9254b8" (UID: "c781c61b-dc25-4a42-ac30-7838bb9254b8"). InnerVolumeSpecName "kube-api-access-rpnbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:34:17.335648 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.335594 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-metrics-certs" pod "c781c61b-dc25-4a42-ac30-7838bb9254b8" (UID: "c781c61b-dc25-4a42-ac30-7838bb9254b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:34:17.396519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.394481 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c781c61b-dc25-4a42-ac30-7838bb9254b8-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:17.396519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.394521 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:17.396519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.394538 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpnbh\" (UniqueName: \"kubernetes.io/projected/c781c61b-dc25-4a42-ac30-7838bb9254b8-kube-api-access-rpnbh\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:17.396519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.394552 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c781c61b-dc25-4a42-ac30-7838bb9254b8-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:34:17.730755 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.730665 5117 generic.go:356] "Generic (PLEG): container finished" podID="c781c61b-dc25-4a42-ac30-7838bb9254b8" containerID="d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331" exitCode=0 Nov 03 03:34:17.730755 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.730746 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" event={"ID":"c781c61b-dc25-4a42-ac30-7838bb9254b8","Type":"ContainerDied","Data":"d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331"} Nov 03 03:34:17.731003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.730796 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" event={"ID":"c781c61b-dc25-4a42-ac30-7838bb9254b8","Type":"ContainerDied","Data":"676b03146e6d0e91b1ac439adcc5de6382b3f2a0974367d5e6045e9b3651aaa6"} Nov 03 03:34:17.731003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.730810 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt" Nov 03 03:34:17.731003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.730824 5117 scope.go:122] "RemoveContainer" containerID="d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331" Nov 03 03:34:17.749822 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.745098 5117 scope.go:122] "RemoveContainer" containerID="d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331" Nov 03 03:34:17.749822 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:34:17.745503 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331\": container with ID starting with d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331 not found: ID does not exist" containerID="d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331" Nov 03 03:34:17.749822 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.745530 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331"} err="failed to get container status \"d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331\": rpc error: code = NotFound desc = could not find container \"d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331\": container with ID starting with d8dd3d00f3e47a04f089ffb4259ec26bc2d37e046580bb1071664ab8c1064331 not found: ID does not exist" Nov 03 03:34:17.903171 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.903136 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt"] Nov 03 03:34:17.974470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:17.956013 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-372186-64b84b58ff-hs8jt"] Nov 03 03:34:19.090432 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:34:19.090397 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c781c61b-dc25-4a42-ac30-7838bb9254b8" path="/var/lib/kubelet/pods/c781c61b-dc25-4a42-ac30-7838bb9254b8/volumes" Nov 03 03:34:30.014048 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:34:30.014007 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:35:00.011973 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:35:00.011930 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:35:17.394258 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.394218 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c"] Nov 03 03:35:17.394777 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.394484 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" podUID="b49ed2d8-458a-4464-ac2b-50788bd13f40" containerName="maas-api" containerID="cri-o://afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c" gracePeriod=30 Nov 03 03:35:17.722038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.722002 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:35:17.906781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.906750 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-metrics-certs\") pod \"b49ed2d8-458a-4464-ac2b-50788bd13f40\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " Nov 03 03:35:17.906954 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.906790 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-maas-api-tls\") pod \"b49ed2d8-458a-4464-ac2b-50788bd13f40\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " Nov 03 03:35:17.907012 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.906947 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49ed2d8-458a-4464-ac2b-50788bd13f40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49ed2d8-458a-4464-ac2b-50788bd13f40-openshift-service-ca\") pod \"b49ed2d8-458a-4464-ac2b-50788bd13f40\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " Nov 03 03:35:17.907192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.907176 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49ed2d8-458a-4464-ac2b-50788bd13f40-kube-api-access-l6ncd\" (UniqueName: \"kubernetes.io/projected/b49ed2d8-458a-4464-ac2b-50788bd13f40-kube-api-access-l6ncd\") pod \"b49ed2d8-458a-4464-ac2b-50788bd13f40\" (UID: \"b49ed2d8-458a-4464-ac2b-50788bd13f40\") " Nov 03 03:35:17.907307 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.907273 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49ed2d8-458a-4464-ac2b-50788bd13f40-openshift-service-ca" pod "b49ed2d8-458a-4464-ac2b-50788bd13f40" (UID: "b49ed2d8-458a-4464-ac2b-50788bd13f40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:35:17.907678 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.907612 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49ed2d8-458a-4464-ac2b-50788bd13f40-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:17.909203 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.909182 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-maas-api-tls" pod "b49ed2d8-458a-4464-ac2b-50788bd13f40" (UID: "b49ed2d8-458a-4464-ac2b-50788bd13f40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:35:17.909298 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.909235 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-metrics-certs" pod "b49ed2d8-458a-4464-ac2b-50788bd13f40" (UID: "b49ed2d8-458a-4464-ac2b-50788bd13f40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:35:17.909364 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:17.909349 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49ed2d8-458a-4464-ac2b-50788bd13f40-kube-api-access-l6ncd" pod "b49ed2d8-458a-4464-ac2b-50788bd13f40" (UID: "b49ed2d8-458a-4464-ac2b-50788bd13f40"). InnerVolumeSpecName "kube-api-access-l6ncd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:35:18.009377 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.009280 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6ncd\" (UniqueName: \"kubernetes.io/projected/b49ed2d8-458a-4464-ac2b-50788bd13f40-kube-api-access-l6ncd\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:18.009377 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.009317 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:18.009377 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.009332 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49ed2d8-458a-4464-ac2b-50788bd13f40-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:35:18.199411 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.199381 5117 generic.go:356] "Generic (PLEG): container finished" podID="b49ed2d8-458a-4464-ac2b-50788bd13f40" containerID="afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c" exitCode=0 Nov 03 03:35:18.199633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.199438 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" Nov 03 03:35:18.199633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.199480 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" event={"ID":"b49ed2d8-458a-4464-ac2b-50788bd13f40","Type":"ContainerDied","Data":"afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c"} Nov 03 03:35:18.199633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.199543 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c" event={"ID":"b49ed2d8-458a-4464-ac2b-50788bd13f40","Type":"ContainerDied","Data":"c57e1171d6358c9c6e188187297762513ad148bf16822c9cc29d4b59f8ac7c89"} Nov 03 03:35:18.199633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.199573 5117 scope.go:122] "RemoveContainer" containerID="afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c" Nov 03 03:35:18.210833 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.210810 5117 scope.go:122] "RemoveContainer" containerID="afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c" Nov 03 03:35:18.211140 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:35:18.211112 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c\": container with ID starting with afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c not found: ID does not exist" containerID="afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c" Nov 03 03:35:18.211209 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.211154 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c"} err="failed to get container status \"afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c\": rpc error: code = NotFound desc = could not find container \"afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c\": container with ID starting with afd0a5e13080e0a7d1c5ab0ed2b28afcea1664f2df2486a135cd980e9d6d052c not found: ID does not exist" Nov 03 03:35:18.382294 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.382260 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c"] Nov 03 03:35:18.387785 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:18.387422 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-211ed3-87677f65f-t9j7c"] Nov 03 03:35:19.089998 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:19.089966 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49ed2d8-458a-4464-ac2b-50788bd13f40" path="/var/lib/kubelet/pods/b49ed2d8-458a-4464-ac2b-50788bd13f40/volumes" Nov 03 03:35:30.011670 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:35:30.011633 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:35:51.573948 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.573919 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm"] Nov 03 03:35:51.574463 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574433 5117 cpu_manager.go:414] "containerMap: removing container" podUID="ecd06f6a-072b-44da-946d-0e590525b929" containerName="maas-api" Nov 03 03:35:51.574463 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574458 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd06f6a-072b-44da-946d-0e590525b929" containerName="maas-api" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574471 5117 cpu_manager.go:414] "containerMap: removing container" podUID="32636981-4bf4-46f0-8c56-54738881439a" containerName="test-curl-7116-98694c" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574476 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32636981-4bf4-46f0-8c56-54738881439a" containerName="test-curl-7116-98694c" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574488 5117 cpu_manager.go:414] "containerMap: removing container" podUID="c781c61b-dc25-4a42-ac30-7838bb9254b8" containerName="maas-api" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574493 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c781c61b-dc25-4a42-ac30-7838bb9254b8" containerName="maas-api" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574503 5117 cpu_manager.go:414] "containerMap: removing container" podUID="99d6a2d9-868b-4049-ad63-254b83d5a797" containerName="main" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574507 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d6a2d9-868b-4049-ad63-254b83d5a797" containerName="main" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574514 5117 cpu_manager.go:414] "containerMap: removing container" podUID="b49ed2d8-458a-4464-ac2b-50788bd13f40" containerName="maas-api" Nov 03 03:35:51.574549 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574519 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49ed2d8-458a-4464-ac2b-50788bd13f40" containerName="maas-api" Nov 03 03:35:51.574792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574623 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd06f6a-072b-44da-946d-0e590525b929" containerName="maas-api" Nov 03 03:35:51.574792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574631 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="32636981-4bf4-46f0-8c56-54738881439a" containerName="test-curl-7116-98694c" Nov 03 03:35:51.574792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574640 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="c781c61b-dc25-4a42-ac30-7838bb9254b8" containerName="maas-api" Nov 03 03:35:51.574792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574646 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d6a2d9-868b-4049-ad63-254b83d5a797" containerName="main" Nov 03 03:35:51.574792 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.574651 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49ed2d8-458a-4464-ac2b-50788bd13f40" containerName="maas-api" Nov 03 03:35:51.577871 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.577849 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.656478 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.656425 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lzs\" (UniqueName: \"kubernetes.io/projected/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-kube-api-access-96lzs\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.656660 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.656560 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.656660 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.656619 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.656759 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.656667 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.684575 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.684550 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b6ef2c-serving-cert\"" Nov 03 03:35:51.687688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.687667 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm"] Nov 03 03:35:51.757698 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.757668 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lzs\" (UniqueName: \"kubernetes.io/projected/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-kube-api-access-96lzs\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.757860 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.757732 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.757860 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.757775 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.757860 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.757792 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.758044 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:35:51.757914 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-serving-cert: secret "maas-api-e2e-ipp-b-b6ef2c-serving-cert" not found Nov 03 03:35:51.758044 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:35:51.757967 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls podName:217f632e-a4c1-42bf-9c0d-adb422c9a8e2 nodeName:}" failed. No retries permitted until 2026-11-03 03:35:52.257952859 +0000 UTC m=+2087.288551389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls") pod "maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" (UID: "217f632e-a4c1-42bf-9c0d-adb422c9a8e2") : secret "maas-api-e2e-ipp-b-b6ef2c-serving-cert" not found Nov 03 03:35:51.758525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.758499 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.760157 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.760133 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:51.820371 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:51.820330 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lzs\" (UniqueName: \"kubernetes.io/projected/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-kube-api-access-96lzs\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:52.263557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:52.263522 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:52.266082 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:52.266062 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:52.492699 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:52.492668 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:52.871356 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:52.871325 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm"] Nov 03 03:35:52.871831 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:35:52.871809 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217f632e_a4c1_42bf_9c0d_adb422c9a8e2.slice/crio-b32d36ff660961ff5a2555a1a68cfc0d58fadbb4854c93900d72d662cee51eb0 WatchSource:0}: Error finding container b32d36ff660961ff5a2555a1a68cfc0d58fadbb4854c93900d72d662cee51eb0: Status 404 returned error can't find the container with id b32d36ff660961ff5a2555a1a68cfc0d58fadbb4854c93900d72d662cee51eb0 Nov 03 03:35:53.460996 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:53.460961 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" event={"ID":"217f632e-a4c1-42bf-9c0d-adb422c9a8e2","Type":"ContainerStarted","Data":"b32d36ff660961ff5a2555a1a68cfc0d58fadbb4854c93900d72d662cee51eb0"} Nov 03 03:35:54.469505 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:54.469465 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" event={"ID":"217f632e-a4c1-42bf-9c0d-adb422c9a8e2","Type":"ContainerStarted","Data":"0cb31098e6bf641c7d42c3b431ca0486681a9b972be0884e4158166c9d1c434a"} Nov 03 03:35:54.469901 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:54.469545 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:35:54.593821 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:35:54.593766 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" podStartSLOduration=2.723049419 podStartE2EDuration="3.59375367s" totalImagesPullingTime="870.704251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:54.591626504 +0000 UTC m=+2089.622225047" watchObservedRunningTime="2026-11-03 03:35:54.59375367 +0000 UTC m=+2089.624352208" Nov 03 03:36:00.011378 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:36:00.011339 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:36:00.481539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:00.481510 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:36:06.693007 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:06.692952 5117 scope.go:122] "RemoveContainer" containerID="5403f32b0e155a6c4a8936466fdb2c2bc445167be8aa3e318ea0a38534934890" Nov 03 03:36:07.435343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:07.435269 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:36:07.439336 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:07.439308 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:36:09.403816 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:09.403778 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt"] Nov 03 03:36:09.404277 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:09.403981 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" containerID="cri-o://c5a1106ba3948367f603db69355bbcf9c3d3fa84ca283a6d8d643c0eaae9e375" gracePeriod=30 Nov 03 03:36:13.219361 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:13.219302 5117 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:36:13.220058 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:13.220004 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:36:14.606042 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.606007 5117 generic.go:356] "Generic (PLEG): container finished" podID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerID="c5a1106ba3948367f603db69355bbcf9c3d3fa84ca283a6d8d643c0eaae9e375" exitCode=0 Nov 03 03:36:14.606430 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.606091 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" event={"ID":"6cecc35b-3e63-46f1-b3e9-2def6166547a","Type":"ContainerDied","Data":"c5a1106ba3948367f603db69355bbcf9c3d3fa84ca283a6d8d643c0eaae9e375"} Nov 03 03:36:14.678931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.678865 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:36:14.770850 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.770814 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-token\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-token\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.770944 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-podinfo\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.770969 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cecc35b-3e63-46f1-b3e9-2def6166547a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cecc35b-3e63-46f1-b3e9-2def6166547a-istiod-ca-cert\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.770996 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-certs\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771023 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-kube-api-access-vcxc2\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-kube-api-access-vcxc2\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771263 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771057 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-socket\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771263 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771110 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-data\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771263 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771226 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-envoy\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771428 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771274 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-credential-socket\") pod \"6cecc35b-3e63-46f1-b3e9-2def6166547a\" (UID: \"6cecc35b-3e63-46f1-b3e9-2def6166547a\") " Nov 03 03:36:14.771428 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771357 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-certs" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:14.771592 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771412 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cecc35b-3e63-46f1-b3e9-2def6166547a-istiod-ca-cert" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:36:14.771649 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771588 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-data" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:14.771816 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771774 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-socket" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:14.771929 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.771894 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-credential-socket" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:14.772217 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.772195 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.772261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.772223 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.772261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.772238 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cecc35b-3e63-46f1-b3e9-2def6166547a-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.772261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.772253 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.772362 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.772268 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.773431 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.773411 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-podinfo" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:36:14.773607 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.773584 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-envoy" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:14.773845 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.773825 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-token" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:36:14.773892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.773855 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-kube-api-access-vcxc2" pod "6cecc35b-3e63-46f1-b3e9-2def6166547a" (UID: "6cecc35b-3e63-46f1-b3e9-2def6166547a"). InnerVolumeSpecName "kube-api-access-vcxc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:36:14.873036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.873002 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.873036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.873031 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxc2\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-kube-api-access-vcxc2\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.873036 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.873043 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:14.873345 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:14.873052 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cecc35b-3e63-46f1-b3e9-2def6166547a-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:15.614166 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:15.614139 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" Nov 03 03:36:15.614640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:15.614132 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt" event={"ID":"6cecc35b-3e63-46f1-b3e9-2def6166547a","Type":"ContainerDied","Data":"31ad88367df2f006d45ed7756632a018c46926c5d79cbd9298de3a246b64e24e"} Nov 03 03:36:15.614640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:15.614286 5117 scope.go:122] "RemoveContainer" containerID="c5a1106ba3948367f603db69355bbcf9c3d3fa84ca283a6d8d643c0eaae9e375" Nov 03 03:36:15.834177 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:15.834137 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt"] Nov 03 03:36:15.884891 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:15.884812 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-nf2gt"] Nov 03 03:36:17.091290 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:17.091249 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" path="/var/lib/kubelet/pods/6cecc35b-3e63-46f1-b3e9-2def6166547a/volumes" Nov 03 03:36:29.316028 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.315998 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg"] Nov 03 03:36:29.316593 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.316579 5117 cpu_manager.go:414] "containerMap: removing container" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" Nov 03 03:36:29.316653 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.316595 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" Nov 03 03:36:29.316725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.316716 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cecc35b-3e63-46f1-b3e9-2def6166547a" containerName="istio-proxy" Nov 03 03:36:29.320437 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.320418 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.374634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.374602 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a6b9ff-serving-cert\"" Nov 03 03:36:29.428617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.428583 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg"] Nov 03 03:36:29.457546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.457511 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.457770 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.457748 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.457951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.457927 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-metrics-certs\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.458127 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.458108 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7h5\" (UniqueName: \"kubernetes.io/projected/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-kube-api-access-rm7h5\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.559993 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.559952 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7h5\" (UniqueName: \"kubernetes.io/projected/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-kube-api-access-rm7h5\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.560166 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.560114 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.560166 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.560158 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.560298 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.560216 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-metrics-certs\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.560298 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:36:29.560264 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-serving-cert: secret "maas-api-e2e-ipp-a-a6b9ff-serving-cert" not found Nov 03 03:36:29.560408 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:36:29.560356 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls podName:970bce3e-7dd5-4039-b7d8-27e8b3b67a99 nodeName:}" failed. No retries permitted until 2026-11-03 03:36:30.060334028 +0000 UTC m=+2125.090932560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls") pod "maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" (UID: "970bce3e-7dd5-4039-b7d8-27e8b3b67a99") : secret "maas-api-e2e-ipp-a-a6b9ff-serving-cert" not found Nov 03 03:36:29.561286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.561257 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.591183 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.586043 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-metrics-certs\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:29.626169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:29.626137 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7h5\" (UniqueName: \"kubernetes.io/projected/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-kube-api-access-rm7h5\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:30.012303 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:36:30.012209 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:36:30.066060 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:30.066022 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:30.068603 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:30.068580 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:30.233463 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:30.233418 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:30.613048 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:30.613016 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg"] Nov 03 03:36:30.614016 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:36:30.613991 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970bce3e_7dd5_4039_b7d8_27e8b3b67a99.slice/crio-1ede525098bc7c7718063133c67cdc15cf2089ae898cb05b7b77074cc779ba5c WatchSource:0}: Error finding container 1ede525098bc7c7718063133c67cdc15cf2089ae898cb05b7b77074cc779ba5c: Status 404 returned error can't find the container with id 1ede525098bc7c7718063133c67cdc15cf2089ae898cb05b7b77074cc779ba5c Nov 03 03:36:30.710723 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:30.710686 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" event={"ID":"970bce3e-7dd5-4039-b7d8-27e8b3b67a99","Type":"ContainerStarted","Data":"1ede525098bc7c7718063133c67cdc15cf2089ae898cb05b7b77074cc779ba5c"} Nov 03 03:36:31.721168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:31.721131 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" event={"ID":"970bce3e-7dd5-4039-b7d8-27e8b3b67a99","Type":"ContainerStarted","Data":"d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e"} Nov 03 03:36:31.721705 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:31.721349 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:31.841379 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:31.841323 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" podStartSLOduration=1.9764464290000001 podStartE2EDuration="2.841308135s" totalImagesPullingTime="864.861706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:31.840579113 +0000 UTC m=+2126.871177651" watchObservedRunningTime="2026-11-03 03:36:31.841308135 +0000 UTC m=+2126.871906673" Nov 03 03:36:37.733843 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:37.733813 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:36:47.496226 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:47.496188 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954"] Nov 03 03:36:47.496689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:47.496484 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" containerID="cri-o://059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f" gracePeriod=30 Nov 03 03:36:50.498169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:50.498133 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:36:50.498603 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:50.498195 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:36:52.777756 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.777699 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:36:52.866959 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.866925 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-podinfo\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867004 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-kube-api-access-v6ml5\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-kube-api-access-v6ml5\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867045 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-certs\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867126 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-credential-socket\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867185 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-envoy\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867226 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-socket\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867288 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-token\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-token\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867341 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-data\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867416 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istiod-ca-cert\") pod \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\" (UID: \"031c117b-cd8c-4bc5-84b8-68aea1cf5748\") " Nov 03 03:36:52.867494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867392 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-certs" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:52.867651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867490 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-credential-socket" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:52.867651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867590 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-socket" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:52.867836 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867813 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-data" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:52.867945 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867921 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istiod-ca-cert" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:36:52.868080 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.867994 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.868209 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.868173 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.868209 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.868206 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.868419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.868222 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.869783 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.869756 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-envoy" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:36:52.869887 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.869861 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-token" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:36:52.870091 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.870066 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-kube-api-access-v6ml5" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "kube-api-access-v6ml5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:36:52.870194 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.870085 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-podinfo" pod "031c117b-cd8c-4bc5-84b8-68aea1cf5748" (UID: "031c117b-cd8c-4bc5-84b8-68aea1cf5748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:36:52.906762 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.906734 5117 generic.go:356] "Generic (PLEG): container finished" podID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerID="059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f" exitCode=0 Nov 03 03:36:52.906852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.906787 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" Nov 03 03:36:52.906852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.906817 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" event={"ID":"031c117b-cd8c-4bc5-84b8-68aea1cf5748","Type":"ContainerDied","Data":"059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f"} Nov 03 03:36:52.906922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.906872 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954" event={"ID":"031c117b-cd8c-4bc5-84b8-68aea1cf5748","Type":"ContainerDied","Data":"2bf0f1b92758a3e5cd13d24d8ce92368bc27c4a1af5c339a69e629f037291b5a"} Nov 03 03:36:52.906922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.906893 5117 scope.go:122] "RemoveContainer" containerID="059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f" Nov 03 03:36:52.917199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.917182 5117 scope.go:122] "RemoveContainer" containerID="059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f" Nov 03 03:36:52.917494 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:36:52.917467 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f\": container with ID starting with 059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f not found: ID does not exist" containerID="059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f" Nov 03 03:36:52.917553 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.917508 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f"} err="failed to get container status \"059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f\": rpc error: code = NotFound desc = could not find container \"059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f\": container with ID starting with 059f6940022e17a665230440fb95a2e3d8e421c77e7a37072202e72f816c0e3f not found: ID does not exist" Nov 03 03:36:52.969890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.969857 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.969890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.969886 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.969890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.969897 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ml5\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-kube-api-access-v6ml5\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.970202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.969907 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:52.970202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:52.969916 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031c117b-cd8c-4bc5-84b8-68aea1cf5748-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:36:53.057400 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:53.057319 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954"] Nov 03 03:36:53.061613 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:53.061582 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-c4954"] Nov 03 03:36:53.092309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:36:53.092279 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" path="/var/lib/kubelet/pods/031c117b-cd8c-4bc5-84b8-68aea1cf5748/volumes" Nov 03 03:37:00.010479 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:37:00.010425 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:37:02.956937 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:02.956897 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb"] Nov 03 03:37:02.957437 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:02.957123 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" containerID="cri-o://ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344" gracePeriod=30 Nov 03 03:37:03.989464 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:03.989409 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:37:03.989951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:03.989497 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:37:08.231818 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.231745 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:37:08.271669 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271639 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-podinfo\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.271835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271682 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-token\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.271835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271720 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istiod-ca-cert\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.271835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271742 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-data\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.271835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271825 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-kube-api-access-6srjp\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-kube-api-access-6srjp\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.272045 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271862 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-credential-socket\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.272045 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271943 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-certs\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.272045 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.271998 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-socket\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.272216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272055 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-envoy\") pod \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\" (UID: \"0a9dc019-94af-4592-a0a5-3ab0e1682a1e\") " Nov 03 03:37:08.272216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272088 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-data" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:08.272327 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272219 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istiod-ca-cert" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:37:08.272327 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272215 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-credential-socket" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:08.272327 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272309 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-certs" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:08.272516 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272419 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-socket" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:08.272662 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272643 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.272725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272667 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.272725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272684 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.272725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272698 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.272725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.272711 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.274292 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.274253 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-kube-api-access-6srjp" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "kube-api-access-6srjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:08.274416 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.274330 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-envoy" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:37:08.274568 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.274551 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-token" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:37:08.274568 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.274553 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-podinfo" pod "0a9dc019-94af-4592-a0a5-3ab0e1682a1e" (UID: "0a9dc019-94af-4592-a0a5-3ab0e1682a1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:37:08.373931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.373885 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.373931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.373921 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.373931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.373935 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:08.374186 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:08.373948 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6srjp\" (UniqueName: \"kubernetes.io/projected/0a9dc019-94af-4592-a0a5-3ab0e1682a1e-kube-api-access-6srjp\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:37:09.023264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.023223 5117 generic.go:356] "Generic (PLEG): container finished" podID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerID="ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344" exitCode=0 Nov 03 03:37:09.023630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.023317 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" Nov 03 03:37:09.023630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.023315 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" event={"ID":"0a9dc019-94af-4592-a0a5-3ab0e1682a1e","Type":"ContainerDied","Data":"ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344"} Nov 03 03:37:09.023630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.023558 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb" event={"ID":"0a9dc019-94af-4592-a0a5-3ab0e1682a1e","Type":"ContainerDied","Data":"561bd09b385553060aed1c0a55c70506345d950fc57e086e2b2f7496d5362cf9"} Nov 03 03:37:09.023630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.023630 5117 scope.go:122] "RemoveContainer" containerID="ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344" Nov 03 03:37:09.059172 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.059148 5117 scope.go:122] "RemoveContainer" containerID="ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344" Nov 03 03:37:09.059629 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:37:09.059591 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344\": container with ID starting with ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344 not found: ID does not exist" containerID="ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344" Nov 03 03:37:09.059783 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.059633 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344"} err="failed to get container status \"ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344\": rpc error: code = NotFound desc = could not find container \"ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344\": container with ID starting with ddc53e88f20a46c8a1b0e94e115aa1d5400e450393cad13ce1b0878fbac06344 not found: ID does not exist" Nov 03 03:37:09.194940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.194909 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb"] Nov 03 03:37:09.244651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:09.244617 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qp2pb"] Nov 03 03:37:11.090077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:11.090043 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" path="/var/lib/kubelet/pods/0a9dc019-94af-4592-a0a5-3ab0e1682a1e/volumes" Nov 03 03:37:27.360341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.360304 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth"] Nov 03 03:37:27.360875 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.360861 5117 cpu_manager.go:414] "containerMap: removing container" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" Nov 03 03:37:27.360875 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.360876 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" Nov 03 03:37:27.360962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.360898 5117 cpu_manager.go:414] "containerMap: removing container" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" Nov 03 03:37:27.360962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.360903 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" Nov 03 03:37:27.361023 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.361000 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c117b-cd8c-4bc5-84b8-68aea1cf5748" containerName="istio-proxy" Nov 03 03:37:27.361023 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.361011 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9dc019-94af-4592-a0a5-3ab0e1682a1e" containerName="istio-proxy" Nov 03 03:37:27.364382 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.364363 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.420109 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.420072 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d6e985-serving-cert\"" Nov 03 03:37:27.420723 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.420677 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.420866 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.420780 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-metrics-certs\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.420866 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.420818 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3360b775-e435-4aea-bd7d-ba099a17ff13-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.421014 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.420993 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qjq\" (UniqueName: \"kubernetes.io/projected/3360b775-e435-4aea-bd7d-ba099a17ff13-kube-api-access-76qjq\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.473891 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.473856 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth"] Nov 03 03:37:27.521997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.521966 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.522167 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.522019 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-metrics-certs\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.522167 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.522041 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3360b775-e435-4aea-bd7d-ba099a17ff13-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.522167 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:37:27.522135 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-serving-cert: secret "maas-api-e2e-ipp-a-d6e985-serving-cert" not found Nov 03 03:37:27.522318 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.522173 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qjq\" (UniqueName: \"kubernetes.io/projected/3360b775-e435-4aea-bd7d-ba099a17ff13-kube-api-access-76qjq\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.522318 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:37:27.522207 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls podName:3360b775-e435-4aea-bd7d-ba099a17ff13 nodeName:}" failed. No retries permitted until 2026-11-03 03:37:28.022185786 +0000 UTC m=+2183.052784316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls") pod "maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" (UID: "3360b775-e435-4aea-bd7d-ba099a17ff13") : secret "maas-api-e2e-ipp-a-d6e985-serving-cert" not found Nov 03 03:37:27.522847 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.522825 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3360b775-e435-4aea-bd7d-ba099a17ff13-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.524698 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.524669 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-metrics-certs\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:27.584584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:27.584549 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qjq\" (UniqueName: \"kubernetes.io/projected/3360b775-e435-4aea-bd7d-ba099a17ff13-kube-api-access-76qjq\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:28.030289 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:28.030245 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:28.033009 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:28.032977 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:28.278185 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:28.278154 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:28.656610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:28.656196 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth"] Nov 03 03:37:28.657025 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:37:28.656861 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3360b775_e435_4aea_bd7d_ba099a17ff13.slice/crio-2aaf8e9f822df4ec72d50b5a699e1cee52fa00b93bef3ea0ff5df33959e888a4 WatchSource:0}: Error finding container 2aaf8e9f822df4ec72d50b5a699e1cee52fa00b93bef3ea0ff5df33959e888a4: Status 404 returned error can't find the container with id 2aaf8e9f822df4ec72d50b5a699e1cee52fa00b93bef3ea0ff5df33959e888a4 Nov 03 03:37:29.154292 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:29.154254 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" event={"ID":"3360b775-e435-4aea-bd7d-ba099a17ff13","Type":"ContainerStarted","Data":"2aaf8e9f822df4ec72d50b5a699e1cee52fa00b93bef3ea0ff5df33959e888a4"} Nov 03 03:37:30.011830 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:37:30.011782 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:37:30.163079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:30.163042 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" event={"ID":"3360b775-e435-4aea-bd7d-ba099a17ff13","Type":"ContainerStarted","Data":"a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd"} Nov 03 03:37:30.163257 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:30.163173 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:30.284876 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:30.284779 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" podStartSLOduration=2.400168468 podStartE2EDuration="3.284766326s" totalImagesPullingTime="884.597858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:30.282088332 +0000 UTC m=+2185.312686875" watchObservedRunningTime="2026-11-03 03:37:30.284766326 +0000 UTC m=+2185.315364864" Nov 03 03:37:36.174490 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:36.174427 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:37:55.161763 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.161680 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5"] Nov 03 03:37:55.165987 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.165969 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.225538 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.225500 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5"] Nov 03 03:37:55.271310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.271275 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8200d4-serving-cert\"" Nov 03 03:37:55.315840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.315806 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-metrics-certs\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.316030 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.315921 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.316030 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.316024 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f020e686-6749-4779-a7e4-bed107c795fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.316167 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.316102 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fznrx\" (UniqueName: \"kubernetes.io/projected/f020e686-6749-4779-a7e4-bed107c795fb-kube-api-access-fznrx\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.417348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.417246 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-metrics-certs\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.417348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.417343 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.417625 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.417433 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f020e686-6749-4779-a7e4-bed107c795fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.417625 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.417561 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fznrx\" (UniqueName: \"kubernetes.io/projected/f020e686-6749-4779-a7e4-bed107c795fb-kube-api-access-fznrx\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.418211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.418135 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f020e686-6749-4779-a7e4-bed107c795fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.420134 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.420101 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-metrics-certs\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.420252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.420170 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.482521 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.482482 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fznrx\" (UniqueName: \"kubernetes.io/projected/f020e686-6749-4779-a7e4-bed107c795fb-kube-api-access-fznrx\") pod \"maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:55.779272 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:55.779172 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:56.165245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:56.165214 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5"] Nov 03 03:37:56.165920 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:37:56.165893 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf020e686_6749_4779_a7e4_bed107c795fb.slice/crio-3e6fa66a0e409a6126c9b068d7626f962b2d8939ef29d3b83cfd5f5a985542c4 WatchSource:0}: Error finding container 3e6fa66a0e409a6126c9b068d7626f962b2d8939ef29d3b83cfd5f5a985542c4: Status 404 returned error can't find the container with id 3e6fa66a0e409a6126c9b068d7626f962b2d8939ef29d3b83cfd5f5a985542c4 Nov 03 03:37:56.167365 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:56.167348 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:37:56.354408 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:56.354368 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" event={"ID":"f020e686-6749-4779-a7e4-bed107c795fb","Type":"ContainerStarted","Data":"3e6fa66a0e409a6126c9b068d7626f962b2d8939ef29d3b83cfd5f5a985542c4"} Nov 03 03:37:57.363190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:57.363098 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" event={"ID":"f020e686-6749-4779-a7e4-bed107c795fb","Type":"ContainerStarted","Data":"da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b"} Nov 03 03:37:57.363640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:57.363186 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:37:57.486547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:37:57.486491 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" podStartSLOduration=1.589834857 podStartE2EDuration="2.486471433s" totalImagesPullingTime="896.636576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:57.485528871 +0000 UTC m=+2212.516127411" watchObservedRunningTime="2026-11-03 03:37:57.486471433 +0000 UTC m=+2212.517069972" Nov 03 03:38:00.011863 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:38:00.011819 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:38:03.374419 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:03.374392 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:38:03.425813 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:03.425782 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m"] Nov 03 03:38:03.425995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:03.425975 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" containerID="cri-o://ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2" gracePeriod=30 Nov 03 03:38:06.047524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:06.047474 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:38:06.047943 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:06.047563 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:38:08.705817 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.705751 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:38:08.789181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789141 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istiod-ca-cert\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789185 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-podinfo\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789211 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-kube-api-access-9597z\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-kube-api-access-9597z\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789241 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-token\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-token\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789268 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-certs\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789330 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-socket\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789413 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-credential-socket\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789745 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789517 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-envoy\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789745 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789544 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-data\") pod \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\" (UID: \"c8841cb2-59ba-47ef-921c-9ed8105c99ca\") " Nov 03 03:38:08.789745 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789650 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istiod-ca-cert" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:38:08.789903 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789747 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-certs" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:38:08.789903 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789767 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-credential-socket" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:38:08.790013 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.789986 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-data" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:38:08.790144 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.790120 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.790242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.790068 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-socket" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:38:08.790296 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.790222 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.790296 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.790286 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.790396 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.790304 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.791831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.791806 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-envoy" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:38:08.791973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.791951 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-kube-api-access-9597z" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "kube-api-access-9597z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:38:08.792213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.792187 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-podinfo" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:38:08.792316 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.792224 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-token" pod "c8841cb2-59ba-47ef-921c-9ed8105c99ca" (UID: "c8841cb2-59ba-47ef-921c-9ed8105c99ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:38:08.891689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.891652 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.891689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.891684 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9597z\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-kube-api-access-9597z\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.891689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.891695 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.892015 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.891704 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:08.892015 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:08.891715 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8841cb2-59ba-47ef-921c-9ed8105c99ca-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:38:09.448480 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.448335 5117 generic.go:356] "Generic (PLEG): container finished" podID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerID="ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2" exitCode=0 Nov 03 03:38:09.448659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.448536 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" event={"ID":"c8841cb2-59ba-47ef-921c-9ed8105c99ca","Type":"ContainerDied","Data":"ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2"} Nov 03 03:38:09.448659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.448625 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" event={"ID":"c8841cb2-59ba-47ef-921c-9ed8105c99ca","Type":"ContainerDied","Data":"45d89a2c9b790f22a7bfe31b8f0f54a5b772e7f7ffc4e6fe4c8960e44b88ff0a"} Nov 03 03:38:09.448767 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.448646 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m" Nov 03 03:38:09.449141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.448667 5117 scope.go:122] "RemoveContainer" containerID="ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2" Nov 03 03:38:09.459786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.459766 5117 scope.go:122] "RemoveContainer" containerID="ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2" Nov 03 03:38:09.460019 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:38:09.460000 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2\": container with ID starting with ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2 not found: ID does not exist" containerID="ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2" Nov 03 03:38:09.460092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.460042 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2"} err="failed to get container status \"ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2\": rpc error: code = NotFound desc = could not find container \"ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2\": container with ID starting with ce2d7a98dbf4cb9e1b2c9fdd50527a747c06c131fc072d71e274714a183978e2 not found: ID does not exist" Nov 03 03:38:09.668719 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.668688 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m"] Nov 03 03:38:09.719874 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:09.719794 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-5bd5m"] Nov 03 03:38:11.090502 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:38:11.090471 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" path="/var/lib/kubelet/pods/c8841cb2-59ba-47ef-921c-9ed8105c99ca/volumes" Nov 03 03:38:30.011596 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:38:30.011558 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:39:00.011943 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:39:00.011899 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:39:30.012263 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:39:30.012177 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:39:37.906120 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:37.906082 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf"] Nov 03 03:39:37.906619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:37.906290 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" containerID="cri-o://b16ae3d2ae8dda2c56fd3766e7eaec7050e675d0bf1956c2f24e3f115d9b14f7" gracePeriod=30 Nov 03 03:39:43.034499 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.034469 5117 generic.go:356] "Generic (PLEG): container finished" podID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerID="b16ae3d2ae8dda2c56fd3766e7eaec7050e675d0bf1956c2f24e3f115d9b14f7" exitCode=0 Nov 03 03:39:43.034883 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.034552 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" event={"ID":"5ad92d0c-e8da-4e5a-90db-ea62ccd20022","Type":"ContainerDied","Data":"b16ae3d2ae8dda2c56fd3766e7eaec7050e675d0bf1956c2f24e3f115d9b14f7"} Nov 03 03:39:43.178234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.178174 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:39:43.254225 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254197 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-socket\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254373 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254244 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-data\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254373 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254282 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-kube-api-access-2tjvd\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-kube-api-access-2tjvd\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254373 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254341 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-token\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-token\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254374 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-podinfo\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254438 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-credential-socket\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254506 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-socket" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:43.254617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254574 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-data" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:43.254838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254619 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istiod-ca-cert\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254663 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-envoy\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254719 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-certs\") pod \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\" (UID: \"5ad92d0c-e8da-4e5a-90db-ea62ccd20022\") " Nov 03 03:39:43.254838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254778 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-credential-socket" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:43.255153 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.254918 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istiod-ca-cert" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:43.255153 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.255080 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-certs" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:43.255430 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.255408 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.255547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.255437 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.255547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.255470 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.255547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.255484 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.255547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.255499 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.256842 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.256815 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-envoy" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:43.256932 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.256909 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-podinfo" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:39:43.256973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.256949 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-kube-api-access-2tjvd" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "kube-api-access-2tjvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:43.257009 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.256977 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-token" pod "5ad92d0c-e8da-4e5a-90db-ea62ccd20022" (UID: "5ad92d0c-e8da-4e5a-90db-ea62ccd20022"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:43.357336 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.357303 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.357336 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.357338 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.357567 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.357352 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjvd\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-kube-api-access-2tjvd\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:43.357567 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:43.357370 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad92d0c-e8da-4e5a-90db-ea62ccd20022-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:44.042874 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:44.042845 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" Nov 03 03:39:44.043816 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:44.042843 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf" event={"ID":"5ad92d0c-e8da-4e5a-90db-ea62ccd20022","Type":"ContainerDied","Data":"7b0f68e3ff34834dac30d4028239e76ae512cc3175f1daf3c08857fefec012d9"} Nov 03 03:39:44.043816 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:44.042985 5117 scope.go:122] "RemoveContainer" containerID="b16ae3d2ae8dda2c56fd3766e7eaec7050e675d0bf1956c2f24e3f115d9b14f7" Nov 03 03:39:44.263178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:44.263142 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf"] Nov 03 03:39:44.314298 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:44.314263 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-d7cbf"] Nov 03 03:39:45.092911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:45.092866 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" path="/var/lib/kubelet/pods/5ad92d0c-e8da-4e5a-90db-ea62ccd20022/volumes" Nov 03 03:39:48.890300 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:48.890262 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9"] Nov 03 03:39:48.890726 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:48.890482 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" containerID="cri-o://3432accbf8ead2cc98912757da4533d04389863fb0a3dba3dd9dacc80434f6ed" gracePeriod=30 Nov 03 03:39:51.123408 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:51.123373 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:39:51.123861 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:51.123458 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:39:54.152690 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.152656 5117 generic.go:356] "Generic (PLEG): container finished" podID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerID="3432accbf8ead2cc98912757da4533d04389863fb0a3dba3dd9dacc80434f6ed" exitCode=0 Nov 03 03:39:54.153143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.152704 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" event={"ID":"27a34e4e-ac22-4381-8c20-4d86850cebe0","Type":"ContainerDied","Data":"3432accbf8ead2cc98912757da4533d04389863fb0a3dba3dd9dacc80434f6ed"} Nov 03 03:39:54.165900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.165870 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:39:54.280333 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280305 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-certs\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280362 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a34e4e-ac22-4381-8c20-4d86850cebe0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a34e4e-ac22-4381-8c20-4d86850cebe0-istiod-ca-cert\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280409 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-credential-socket\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280428 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-podinfo\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280524 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-envoy\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280574 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-socket\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280585 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-certs" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:54.280712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280617 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-token\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-token\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280671 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-kube-api-access-k622t\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-kube-api-access-k622t\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280709 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-data\") pod \"27a34e4e-ac22-4381-8c20-4d86850cebe0\" (UID: \"27a34e4e-ac22-4381-8c20-4d86850cebe0\") " Nov 03 03:39:54.280981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280792 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-credential-socket" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:54.280981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280832 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-socket" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:54.280981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.280868 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a34e4e-ac22-4381-8c20-4d86850cebe0-istiod-ca-cert" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:39:54.281187 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.281166 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-data" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:54.281348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.281324 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.281348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.281349 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.281546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.281364 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.281546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.281377 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.281546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.281390 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a34e4e-ac22-4381-8c20-4d86850cebe0-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.282969 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.282944 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-token" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:54.283078 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.282989 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-podinfo" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:39:54.283318 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.283304 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-envoy" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:39:54.283432 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.283412 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-kube-api-access-k622t" pod "27a34e4e-ac22-4381-8c20-4d86850cebe0" (UID: "27a34e4e-ac22-4381-8c20-4d86850cebe0"). InnerVolumeSpecName "kube-api-access-k622t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:39:54.382597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.382552 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.382597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.382592 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.382597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.382602 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k622t\" (UniqueName: \"kubernetes.io/projected/27a34e4e-ac22-4381-8c20-4d86850cebe0-kube-api-access-k622t\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:54.382597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:54.382613 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a34e4e-ac22-4381-8c20-4d86850cebe0-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:39:55.160092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:55.160066 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" Nov 03 03:39:55.160092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:55.160074 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9" event={"ID":"27a34e4e-ac22-4381-8c20-4d86850cebe0","Type":"ContainerDied","Data":"3d1c36fad63ac21f81f0df82c324467f03331d5046e859e352b661cc2ec801b5"} Nov 03 03:39:55.160588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:55.160139 5117 scope.go:122] "RemoveContainer" containerID="3432accbf8ead2cc98912757da4533d04389863fb0a3dba3dd9dacc80434f6ed" Nov 03 03:39:55.378717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:55.378635 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9"] Nov 03 03:39:55.384240 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:55.384214 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-7dgx9"] Nov 03 03:39:56.132464 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:56.132415 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg"] Nov 03 03:39:56.132728 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:56.132700 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" containerID="cri-o://90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741" gracePeriod=30 Nov 03 03:39:57.090675 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:39:57.090636 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" path="/var/lib/kubelet/pods/27a34e4e-ac22-4381-8c20-4d86850cebe0/volumes" Nov 03 03:40:00.010286 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:40:00.010252 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:40:01.434825 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.434799 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:40:01.463787 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.463758 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e875d1-b7e9-45f2-8d30-498be58ac927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e875d1-b7e9-45f2-8d30-498be58ac927-istiod-ca-cert\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.463988 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.463796 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-token\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-token\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.463988 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.463862 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-kube-api-access-884b5\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-kube-api-access-884b5\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.463988 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.463910 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-socket\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.463988 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.463942 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-certs\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.464251 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464087 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-envoy\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.464251 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464211 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-data\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.464362 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464215 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e875d1-b7e9-45f2-8d30-498be58ac927-istiod-ca-cert" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:01.464362 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464303 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-socket" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:40:01.464362 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464253 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-podinfo\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.464550 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464383 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-credential-socket\") pod \"a8e875d1-b7e9-45f2-8d30-498be58ac927\" (UID: \"a8e875d1-b7e9-45f2-8d30-498be58ac927\") " Nov 03 03:40:01.464612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464567 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-data" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:40:01.464612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464589 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-certs" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:40:01.464779 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.464757 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-credential-socket" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:40:01.465090 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.465069 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.465164 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.465097 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.465164 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.465115 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e875d1-b7e9-45f2-8d30-498be58ac927-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.465164 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.465131 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.465164 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.465147 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.466586 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.466558 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-podinfo" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:40:01.466586 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.466574 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-envoy" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:40:01.466756 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.466657 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-token" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:01.466756 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.466743 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-kube-api-access-884b5" pod "a8e875d1-b7e9-45f2-8d30-498be58ac927" (UID: "a8e875d1-b7e9-45f2-8d30-498be58ac927"). InnerVolumeSpecName "kube-api-access-884b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:01.566119 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.566083 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-884b5\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-kube-api-access-884b5\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.566119 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.566113 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.566119 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.566124 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:01.566372 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:01.566134 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e875d1-b7e9-45f2-8d30-498be58ac927-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:02.208986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.208946 5117 generic.go:356] "Generic (PLEG): container finished" podID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerID="90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741" exitCode=0 Nov 03 03:40:02.209178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.209008 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" Nov 03 03:40:02.209178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.209059 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" event={"ID":"a8e875d1-b7e9-45f2-8d30-498be58ac927","Type":"ContainerDied","Data":"90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741"} Nov 03 03:40:02.209178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.209126 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg" event={"ID":"a8e875d1-b7e9-45f2-8d30-498be58ac927","Type":"ContainerDied","Data":"8cc6573e89ffdce6863d18fafd79bdaadfdd85e1ab8a10025dd1dccade8944b6"} Nov 03 03:40:02.209178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.209155 5117 scope.go:122] "RemoveContainer" containerID="90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741" Nov 03 03:40:02.220861 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.220840 5117 scope.go:122] "RemoveContainer" containerID="90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741" Nov 03 03:40:02.221189 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:40:02.221170 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741\": container with ID starting with 90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741 not found: ID does not exist" containerID="90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741" Nov 03 03:40:02.221238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.221200 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741"} err="failed to get container status \"90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741\": rpc error: code = NotFound desc = could not find container \"90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741\": container with ID starting with 90496c8bbce71952e15819052f62ed8487b8b099217c7046437b0edb1f295741 not found: ID does not exist" Nov 03 03:40:02.430136 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.430100 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg"] Nov 03 03:40:02.481468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:02.481356 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-5fmtg"] Nov 03 03:40:03.090246 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:03.090211 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" path="/var/lib/kubelet/pods/a8e875d1-b7e9-45f2-8d30-498be58ac927/volumes" Nov 03 03:40:30.011145 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:40:30.011107 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:40:40.308242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.308160 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm"] Nov 03 03:40:40.308777 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.308393 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" podUID="217f632e-a4c1-42bf-9c0d-adb422c9a8e2" containerName="maas-api" containerID="cri-o://0cb31098e6bf641c7d42c3b431ca0486681a9b972be0884e4158166c9d1c434a" gracePeriod=30 Nov 03 03:40:40.444711 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.444666 5117 generic.go:356] "Generic (PLEG): container finished" podID="217f632e-a4c1-42bf-9c0d-adb422c9a8e2" containerID="0cb31098e6bf641c7d42c3b431ca0486681a9b972be0884e4158166c9d1c434a" exitCode=0 Nov 03 03:40:40.444872 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.444735 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" event={"ID":"217f632e-a4c1-42bf-9c0d-adb422c9a8e2","Type":"ContainerDied","Data":"0cb31098e6bf641c7d42c3b431ca0486681a9b972be0884e4158166c9d1c434a"} Nov 03 03:40:40.476763 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.476715 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" podUID="217f632e-a4c1-42bf-9c0d-adb422c9a8e2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.95:8443/health\": dial tcp 10.132.0.95:8443: connect: connection refused" Nov 03 03:40:40.589670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.589643 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:40:40.601783 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.601759 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-metrics-certs\") pod \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " Nov 03 03:40:40.601908 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.601846 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-openshift-service-ca\") pod \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " Nov 03 03:40:40.601908 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.601889 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls\") pod \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " Nov 03 03:40:40.601995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.601948 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-kube-api-access-96lzs\" (UniqueName: \"kubernetes.io/projected/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-kube-api-access-96lzs\") pod \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\" (UID: \"217f632e-a4c1-42bf-9c0d-adb422c9a8e2\") " Nov 03 03:40:40.602238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.602210 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-openshift-service-ca" pod "217f632e-a4c1-42bf-9c0d-adb422c9a8e2" (UID: "217f632e-a4c1-42bf-9c0d-adb422c9a8e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:40:40.602391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.602375 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:40.604159 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.604134 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-metrics-certs" pod "217f632e-a4c1-42bf-9c0d-adb422c9a8e2" (UID: "217f632e-a4c1-42bf-9c0d-adb422c9a8e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:40:40.604668 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.604645 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls" pod "217f632e-a4c1-42bf-9c0d-adb422c9a8e2" (UID: "217f632e-a4c1-42bf-9c0d-adb422c9a8e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:40:40.604742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.604666 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-kube-api-access-96lzs" pod "217f632e-a4c1-42bf-9c0d-adb422c9a8e2" (UID: "217f632e-a4c1-42bf-9c0d-adb422c9a8e2"). InnerVolumeSpecName "kube-api-access-96lzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:40:40.705438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.705274 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:40.705438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.705320 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:40.705438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:40.705337 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lzs\" (UniqueName: \"kubernetes.io/projected/217f632e-a4c1-42bf-9c0d-adb422c9a8e2-kube-api-access-96lzs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:40:41.454526 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:41.454420 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" event={"ID":"217f632e-a4c1-42bf-9c0d-adb422c9a8e2","Type":"ContainerDied","Data":"b32d36ff660961ff5a2555a1a68cfc0d58fadbb4854c93900d72d662cee51eb0"} Nov 03 03:40:41.454526 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:41.454436 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm" Nov 03 03:40:41.454526 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:41.454515 5117 scope.go:122] "RemoveContainer" containerID="0cb31098e6bf641c7d42c3b431ca0486681a9b972be0884e4158166c9d1c434a" Nov 03 03:40:41.623684 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:41.623648 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm"] Nov 03 03:40:41.674472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:41.674417 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6ef2c-84fdbfbbf6-rhngm"] Nov 03 03:40:43.089964 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:40:43.089932 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="217f632e-a4c1-42bf-9c0d-adb422c9a8e2" path="/var/lib/kubelet/pods/217f632e-a4c1-42bf-9c0d-adb422c9a8e2/volumes" Nov 03 03:41:00.011937 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:41:00.011823 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:41:04.884919 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.884884 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf"] Nov 03 03:41:04.885416 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885403 5117 cpu_manager.go:414] "containerMap: removing container" podUID="217f632e-a4c1-42bf-9c0d-adb422c9a8e2" containerName="maas-api" Nov 03 03:41:04.885512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885421 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="217f632e-a4c1-42bf-9c0d-adb422c9a8e2" containerName="maas-api" Nov 03 03:41:04.885512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885455 5117 cpu_manager.go:414] "containerMap: removing container" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" Nov 03 03:41:04.885512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885463 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" Nov 03 03:41:04.885512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885495 5117 cpu_manager.go:414] "containerMap: removing container" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" Nov 03 03:41:04.885512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885500 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" Nov 03 03:41:04.885512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885509 5117 cpu_manager.go:414] "containerMap: removing container" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" Nov 03 03:41:04.885512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885513 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" Nov 03 03:41:04.885722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885524 5117 cpu_manager.go:414] "containerMap: removing container" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" Nov 03 03:41:04.885722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885531 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" Nov 03 03:41:04.885722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885619 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad92d0c-e8da-4e5a-90db-ea62ccd20022" containerName="istio-proxy" Nov 03 03:41:04.885722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885628 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e875d1-b7e9-45f2-8d30-498be58ac927" containerName="istio-proxy" Nov 03 03:41:04.885722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885634 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8841cb2-59ba-47ef-921c-9ed8105c99ca" containerName="istio-proxy" Nov 03 03:41:04.885722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885640 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="217f632e-a4c1-42bf-9c0d-adb422c9a8e2" containerName="maas-api" Nov 03 03:41:04.885722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.885647 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a34e4e-ac22-4381-8c20-4d86850cebe0" containerName="istio-proxy" Nov 03 03:41:04.888950 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.888930 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:04.988436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.988405 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-metrics-certs\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:04.988591 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.988548 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:04.988632 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.988613 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384db0de-a7df-47e8-8f4d-f7fdb7982434-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:04.988689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.988677 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh692\" (UniqueName: \"kubernetes.io/projected/384db0de-a7df-47e8-8f4d-f7fdb7982434-kube-api-access-jh692\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:04.994486 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.994462 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-754aeb-serving-cert\"" Nov 03 03:41:04.997494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:04.997475 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf"] Nov 03 03:41:05.089301 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.089273 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.089301 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.089308 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384db0de-a7df-47e8-8f4d-f7fdb7982434-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.089604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.089359 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh692\" (UniqueName: \"kubernetes.io/projected/384db0de-a7df-47e8-8f4d-f7fdb7982434-kube-api-access-jh692\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.089604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.089467 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-metrics-certs\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.090142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.090116 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384db0de-a7df-47e8-8f4d-f7fdb7982434-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.091907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.091878 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-metrics-certs\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.142057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.141954 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-754aeb-serving-cert\"" Nov 03 03:41:05.153939 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:41:05.153910 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-serving-cert: secret "maas-api-e2e-praxis-a-754aeb-serving-cert" not found Nov 03 03:41:05.154229 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:41:05.154205 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls podName:384db0de-a7df-47e8-8f4d-f7fdb7982434 nodeName:}" failed. No retries permitted until 2026-11-03 03:41:05.65418148 +0000 UTC m=+2400.684780013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls") pod "maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" (UID: "384db0de-a7df-47e8-8f4d-f7fdb7982434") : secret "maas-api-e2e-praxis-a-754aeb-serving-cert" not found Nov 03 03:41:05.156216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.156187 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh692\" (UniqueName: \"kubernetes.io/projected/384db0de-a7df-47e8-8f4d-f7fdb7982434-kube-api-access-jh692\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.695238 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.695204 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.697856 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.697823 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls\") pod \"maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:05.802091 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:05.802062 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:06.181871 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:06.181843 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf"] Nov 03 03:41:06.182764 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:41:06.182739 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384db0de_a7df_47e8_8f4d_f7fdb7982434.slice/crio-0bd66732d66f228b16bd9eb7f9e8573f22575a6f208988f5f1e2f4b4d843943b WatchSource:0}: Error finding container 0bd66732d66f228b16bd9eb7f9e8573f22575a6f208988f5f1e2f4b4d843943b: Status 404 returned error can't find the container with id 0bd66732d66f228b16bd9eb7f9e8573f22575a6f208988f5f1e2f4b4d843943b Nov 03 03:41:06.606551 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:06.606513 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" event={"ID":"384db0de-a7df-47e8-8f4d-f7fdb7982434","Type":"ContainerStarted","Data":"0bd66732d66f228b16bd9eb7f9e8573f22575a6f208988f5f1e2f4b4d843943b"} Nov 03 03:41:07.479891 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:07.479850 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:41:07.484732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:07.484704 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:41:07.613978 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:07.613940 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" event={"ID":"384db0de-a7df-47e8-8f4d-f7fdb7982434","Type":"ContainerStarted","Data":"748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a"} Nov 03 03:41:07.614172 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:07.614054 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:07.735958 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:07.735841 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" podStartSLOduration=2.861142133 podStartE2EDuration="3.735827358s" totalImagesPullingTime="874.685225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:07.733573796 +0000 UTC m=+2402.764172335" watchObservedRunningTime="2026-11-03 03:41:07.735827358 +0000 UTC m=+2402.766425895" Nov 03 03:41:13.625323 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:13.625287 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:41:20.953426 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:20.953388 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5"] Nov 03 03:41:20.953916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:20.953604 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" podUID="f020e686-6749-4779-a7e4-bed107c795fb" containerName="maas-api" containerID="cri-o://da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b" gracePeriod=30 Nov 03 03:41:21.257813 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.257780 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:41:21.392841 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.392801 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f020e686-6749-4779-a7e4-bed107c795fb-kube-api-access-fznrx\" (UniqueName: \"kubernetes.io/projected/f020e686-6749-4779-a7e4-bed107c795fb-kube-api-access-fznrx\") pod \"f020e686-6749-4779-a7e4-bed107c795fb\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " Nov 03 03:41:21.393218 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.393193 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f020e686-6749-4779-a7e4-bed107c795fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f020e686-6749-4779-a7e4-bed107c795fb-openshift-service-ca\") pod \"f020e686-6749-4779-a7e4-bed107c795fb\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " Nov 03 03:41:21.393474 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.393432 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-maas-api-tls\") pod \"f020e686-6749-4779-a7e4-bed107c795fb\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " Nov 03 03:41:21.393713 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.393696 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-metrics-certs\") pod \"f020e686-6749-4779-a7e4-bed107c795fb\" (UID: \"f020e686-6749-4779-a7e4-bed107c795fb\") " Nov 03 03:41:21.394077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.394052 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f020e686-6749-4779-a7e4-bed107c795fb-openshift-service-ca" pod "f020e686-6749-4779-a7e4-bed107c795fb" (UID: "f020e686-6749-4779-a7e4-bed107c795fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:41:21.394436 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.394409 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f020e686-6749-4779-a7e4-bed107c795fb-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:21.400270 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.400228 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-metrics-certs" pod "f020e686-6749-4779-a7e4-bed107c795fb" (UID: "f020e686-6749-4779-a7e4-bed107c795fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:41:21.405957 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.405923 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f020e686-6749-4779-a7e4-bed107c795fb-kube-api-access-fznrx" pod "f020e686-6749-4779-a7e4-bed107c795fb" (UID: "f020e686-6749-4779-a7e4-bed107c795fb"). InnerVolumeSpecName "kube-api-access-fznrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:21.406070 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.405957 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-maas-api-tls" pod "f020e686-6749-4779-a7e4-bed107c795fb" (UID: "f020e686-6749-4779-a7e4-bed107c795fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:41:21.495693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.495612 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fznrx\" (UniqueName: \"kubernetes.io/projected/f020e686-6749-4779-a7e4-bed107c795fb-kube-api-access-fznrx\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:21.495693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.495649 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:21.495693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.495664 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f020e686-6749-4779-a7e4-bed107c795fb-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:21.707746 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.707715 5117 generic.go:356] "Generic (PLEG): container finished" podID="f020e686-6749-4779-a7e4-bed107c795fb" containerID="da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b" exitCode=0 Nov 03 03:41:21.707911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.707783 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" Nov 03 03:41:21.707911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.707799 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" event={"ID":"f020e686-6749-4779-a7e4-bed107c795fb","Type":"ContainerDied","Data":"da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b"} Nov 03 03:41:21.707911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.707880 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5" event={"ID":"f020e686-6749-4779-a7e4-bed107c795fb","Type":"ContainerDied","Data":"3e6fa66a0e409a6126c9b068d7626f962b2d8939ef29d3b83cfd5f5a985542c4"} Nov 03 03:41:21.707911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.707902 5117 scope.go:122] "RemoveContainer" containerID="da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b" Nov 03 03:41:21.722196 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.722171 5117 scope.go:122] "RemoveContainer" containerID="da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b" Nov 03 03:41:21.722723 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:41:21.722696 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b\": container with ID starting with da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b not found: ID does not exist" containerID="da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b" Nov 03 03:41:21.722818 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.722732 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b"} err="failed to get container status \"da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b\": rpc error: code = NotFound desc = could not find container \"da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b\": container with ID starting with da7d55fdf2762fb745807240ebfed3e36f444d165cf9f23b774b9de9414d805b not found: ID does not exist" Nov 03 03:41:21.877212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.877181 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5"] Nov 03 03:41:21.881168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:21.881146 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8200d4-fc447b978-t8pt5"] Nov 03 03:41:23.091028 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:23.090991 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f020e686-6749-4779-a7e4-bed107c795fb" path="/var/lib/kubelet/pods/f020e686-6749-4779-a7e4-bed107c795fb/volumes" Nov 03 03:41:30.011365 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:41:30.011328 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:41:41.882083 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:41.882053 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp"] Nov 03 03:41:41.882623 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:41.882243 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" containerID="cri-o://c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae" gracePeriod=30 Nov 03 03:41:43.032875 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:43.032835 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:41:43.033251 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:43.032901 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:41:47.145981 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.145922 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:41:47.193781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193395 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-socket\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.193781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193518 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-data\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.193781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193588 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-kube-api-access-mrslx\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-kube-api-access-mrslx\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.193781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193661 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-credential-socket\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.193781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193773 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-certs\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.194160 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193820 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-envoy\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.194160 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193884 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istiod-ca-cert\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.194160 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193942 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-podinfo\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.194160 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.193987 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-token\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-token\") pod \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\" (UID: \"50cc62d3-2b5f-4f34-a2ba-7648a5094228\") " Nov 03 03:41:47.195422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.194407 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-socket" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:47.195422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.194677 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-certs" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:47.195422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.194852 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-data" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:47.195422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.195100 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.195422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.195126 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.195422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.195139 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.198241 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.198061 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-credential-socket" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:47.198763 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.198504 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istiod-ca-cert" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:41:47.203310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.201883 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-podinfo" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:41:47.203310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.203179 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-kube-api-access-mrslx" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "kube-api-access-mrslx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:47.204148 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.204127 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-token" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:41:47.205080 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.205066 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-envoy" pod "50cc62d3-2b5f-4f34-a2ba-7648a5094228" (UID: "50cc62d3-2b5f-4f34-a2ba-7648a5094228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:41:47.295997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.295965 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrslx\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-kube-api-access-mrslx\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.295997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.295994 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.295997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.296005 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.296237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.296014 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.296237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.296023 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.296237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.296031 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc62d3-2b5f-4f34-a2ba-7648a5094228-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:41:47.868247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.868211 5117 generic.go:356] "Generic (PLEG): container finished" podID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerID="c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae" exitCode=0 Nov 03 03:41:47.868433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.868248 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" event={"ID":"50cc62d3-2b5f-4f34-a2ba-7648a5094228","Type":"ContainerDied","Data":"c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae"} Nov 03 03:41:47.868433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.868279 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" Nov 03 03:41:47.868433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.868332 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp" event={"ID":"50cc62d3-2b5f-4f34-a2ba-7648a5094228","Type":"ContainerDied","Data":"bb4cfdc73f6f7fe849a1f6a083620c7f60c6b59c193e1684b80f25fbcb9e083f"} Nov 03 03:41:47.868433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.868362 5117 scope.go:122] "RemoveContainer" containerID="c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae" Nov 03 03:41:47.879076 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.879061 5117 scope.go:122] "RemoveContainer" containerID="c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae" Nov 03 03:41:47.879340 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:41:47.879321 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae\": container with ID starting with c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae not found: ID does not exist" containerID="c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae" Nov 03 03:41:47.879389 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:47.879348 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae"} err="failed to get container status \"c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae\": rpc error: code = NotFound desc = could not find container \"c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae\": container with ID starting with c04e3cf015c807199664572584b966c9a144e44a2123dc748753711b3361b6ae not found: ID does not exist" Nov 03 03:41:48.089101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:48.089064 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp"] Nov 03 03:41:48.139282 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:48.139212 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-btqfp"] Nov 03 03:41:49.090645 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:41:49.090609 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" path="/var/lib/kubelet/pods/50cc62d3-2b5f-4f34-a2ba-7648a5094228/volumes" Nov 03 03:42:00.010889 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:42:00.010843 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:42:02.482616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.482581 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w"] Nov 03 03:42:02.483114 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.483100 5117 cpu_manager.go:414] "containerMap: removing container" podUID="f020e686-6749-4779-a7e4-bed107c795fb" containerName="maas-api" Nov 03 03:42:02.483158 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.483115 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f020e686-6749-4779-a7e4-bed107c795fb" containerName="maas-api" Nov 03 03:42:02.483158 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.483134 5117 cpu_manager.go:414] "containerMap: removing container" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" Nov 03 03:42:02.483158 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.483139 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" Nov 03 03:42:02.483256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.483247 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="f020e686-6749-4779-a7e4-bed107c795fb" containerName="maas-api" Nov 03 03:42:02.483296 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.483259 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cc62d3-2b5f-4f34-a2ba-7648a5094228" containerName="istio-proxy" Nov 03 03:42:02.486879 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.486852 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.592787 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.592754 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b755ee-serving-cert\"" Nov 03 03:42:02.595984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.595958 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w"] Nov 03 03:42:02.668939 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.668894 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56bb8917-cc8f-42ce-a4de-e4e92bccc348-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.669111 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.668955 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.669111 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.669057 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfcs\" (UniqueName: \"kubernetes.io/projected/56bb8917-cc8f-42ce-a4de-e4e92bccc348-kube-api-access-cmfcs\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.669111 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.669094 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-metrics-certs\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.770838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.770735 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56bb8917-cc8f-42ce-a4de-e4e92bccc348-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.770838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.770814 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.771095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.770875 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfcs\" (UniqueName: \"kubernetes.io/projected/56bb8917-cc8f-42ce-a4de-e4e92bccc348-kube-api-access-cmfcs\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.771095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.770910 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-metrics-certs\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.771095 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:42:02.770930 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-serving-cert: secret "maas-api-e2e-praxis-a-b755ee-serving-cert" not found Nov 03 03:42:02.771095 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:42:02.770997 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls podName:56bb8917-cc8f-42ce-a4de-e4e92bccc348 nodeName:}" failed. No retries permitted until 2026-11-03 03:42:03.270978341 +0000 UTC m=+2458.301576870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls") pod "maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" (UID: "56bb8917-cc8f-42ce-a4de-e4e92bccc348") : secret "maas-api-e2e-praxis-a-b755ee-serving-cert" not found Nov 03 03:42:02.771512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.771490 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56bb8917-cc8f-42ce-a4de-e4e92bccc348-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.773720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.773695 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-metrics-certs\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:02.834694 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:02.834659 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfcs\" (UniqueName: \"kubernetes.io/projected/56bb8917-cc8f-42ce-a4de-e4e92bccc348-kube-api-access-cmfcs\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:03.275374 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:03.275331 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:03.278041 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:03.278011 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:03.401319 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:03.401278 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:03.780852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:03.780820 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w"] Nov 03 03:42:03.781883 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:42:03.781856 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bb8917_cc8f_42ce_a4de_e4e92bccc348.slice/crio-5b5e66a03b02f1d42a19f9cf6cab838fae26acddbdb55bcd399a710bcf8f596d WatchSource:0}: Error finding container 5b5e66a03b02f1d42a19f9cf6cab838fae26acddbdb55bcd399a710bcf8f596d: Status 404 returned error can't find the container with id 5b5e66a03b02f1d42a19f9cf6cab838fae26acddbdb55bcd399a710bcf8f596d Nov 03 03:42:03.978184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:03.978149 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" event={"ID":"56bb8917-cc8f-42ce-a4de-e4e92bccc348","Type":"ContainerStarted","Data":"5b5e66a03b02f1d42a19f9cf6cab838fae26acddbdb55bcd399a710bcf8f596d"} Nov 03 03:42:04.985804 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:04.985710 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" event={"ID":"56bb8917-cc8f-42ce-a4de-e4e92bccc348","Type":"ContainerStarted","Data":"3002a88a11cefbd44d7ea7ee163e7a09ed2a67d44ffb931826be65b67eca30fe"} Nov 03 03:42:04.986199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:04.985868 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:05.108391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:05.108336 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" podStartSLOduration=2.280399153 podStartE2EDuration="3.108322879s" totalImagesPullingTime="827.923726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:05.105981773 +0000 UTC m=+2460.136580312" watchObservedRunningTime="2026-11-03 03:42:05.108322879 +0000 UTC m=+2460.138921446" Nov 03 03:42:07.910087 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:07.910053 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55"] Nov 03 03:42:07.910526 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:07.910262 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" containerID="cri-o://b32aa9da98a5ae43f5a3a8e2faa1d370322ca7ccb26c6217314ab79cf96b7df1" gracePeriod=30 Nov 03 03:42:10.996788 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:10.996755 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:42:13.035261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.035226 5117 generic.go:356] "Generic (PLEG): container finished" podID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerID="b32aa9da98a5ae43f5a3a8e2faa1d370322ca7ccb26c6217314ab79cf96b7df1" exitCode=0 Nov 03 03:42:13.035672 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.035292 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" event={"ID":"c555ca6b-db4c-4b18-8a3a-1a279d78556f","Type":"ContainerDied","Data":"b32aa9da98a5ae43f5a3a8e2faa1d370322ca7ccb26c6217314ab79cf96b7df1"} Nov 03 03:42:13.171112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.171055 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:42:13.178296 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178274 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-data\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.178405 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178375 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-certs\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.178475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178407 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-socket\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.178548 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178511 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-envoy\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.178606 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178579 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-data" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:42:13.178606 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178584 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-credential-socket\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.178736 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178718 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-kube-api-access-k6df6\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-kube-api-access-k6df6\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.178803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178726 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-certs" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:42:13.178803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178751 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-socket" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:42:13.178803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178760 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-podinfo\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.178803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178772 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-credential-socket" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:42:13.178803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178788 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-token\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-token\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.179068 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.178827 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istiod-ca-cert\") pod \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\" (UID: \"c555ca6b-db4c-4b18-8a3a-1a279d78556f\") " Nov 03 03:42:13.179566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.179379 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.179566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.179390 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istiod-ca-cert" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:42:13.179566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.179405 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.179566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.179420 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.179566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.179434 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.180967 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.180938 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-envoy" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:42:13.181065 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.181017 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-podinfo" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:42:13.181491 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.181463 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-token" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:13.181588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.181523 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-kube-api-access-k6df6" pod "c555ca6b-db4c-4b18-8a3a-1a279d78556f" (UID: "c555ca6b-db4c-4b18-8a3a-1a279d78556f"). InnerVolumeSpecName "kube-api-access-k6df6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:42:13.280540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.280511 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6df6\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-kube-api-access-k6df6\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.280540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.280540 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.280540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.280555 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.280766 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.280569 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:13.280766 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:13.280582 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c555ca6b-db4c-4b18-8a3a-1a279d78556f-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:42:14.043320 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:14.043291 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" Nov 03 03:42:14.043855 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:14.043325 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55" event={"ID":"c555ca6b-db4c-4b18-8a3a-1a279d78556f","Type":"ContainerDied","Data":"81ddbd914eeb5ee172e704d18bedc2fa8e56238588c25f5dc1934de68f6f6062"} Nov 03 03:42:14.043855 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:14.043392 5117 scope.go:122] "RemoveContainer" containerID="b32aa9da98a5ae43f5a3a8e2faa1d370322ca7ccb26c6217314ab79cf96b7df1" Nov 03 03:42:14.264671 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:14.264633 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55"] Nov 03 03:42:14.319185 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:14.317935 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-7cb55"] Nov 03 03:42:15.090881 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:15.090847 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" path="/var/lib/kubelet/pods/c555ca6b-db4c-4b18-8a3a-1a279d78556f/volumes" Nov 03 03:42:30.011496 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:42:30.011413 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:42:47.284947 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.284911 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w"] Nov 03 03:42:47.285458 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.285426 5117 cpu_manager.go:414] "containerMap: removing container" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" Nov 03 03:42:47.285525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.285469 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" Nov 03 03:42:47.285585 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.285575 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="c555ca6b-db4c-4b18-8a3a-1a279d78556f" containerName="istio-proxy" Nov 03 03:42:47.290033 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.290012 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.345154 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.345109 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-29dd3f-serving-cert\"" Nov 03 03:42:47.349750 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.349722 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc59226-544a-4420-be01-394f2654df9b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.349898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.349798 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.349898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.349873 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8q2\" (UniqueName: \"kubernetes.io/projected/afc59226-544a-4420-be01-394f2654df9b-kube-api-access-5m8q2\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.350005 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.349947 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-metrics-certs\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.398952 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.398922 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w"] Nov 03 03:42:47.451370 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.451332 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc59226-544a-4420-be01-394f2654df9b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.451635 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.451488 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.451635 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.451523 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8q2\" (UniqueName: \"kubernetes.io/projected/afc59226-544a-4420-be01-394f2654df9b-kube-api-access-5m8q2\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.451635 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.451569 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-metrics-certs\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.451857 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:42:47.451688 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-serving-cert: secret "maas-api-e2e-praxis-a-29dd3f-serving-cert" not found Nov 03 03:42:47.451857 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:42:47.451803 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls podName:afc59226-544a-4420-be01-394f2654df9b nodeName:}" failed. No retries permitted until 2026-11-03 03:42:47.951781104 +0000 UTC m=+2502.982379637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls") pod "maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" (UID: "afc59226-544a-4420-be01-394f2654df9b") : secret "maas-api-e2e-praxis-a-29dd3f-serving-cert" not found Nov 03 03:42:47.452585 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.452558 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc59226-544a-4420-be01-394f2654df9b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.454468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.454412 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-metrics-certs\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.516219 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.516183 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8q2\" (UniqueName: \"kubernetes.io/projected/afc59226-544a-4420-be01-394f2654df9b-kube-api-access-5m8q2\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.957633 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.957598 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:47.960124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:47.960097 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:48.203616 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:48.203580 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:48.586429 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:48.586399 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w"] Nov 03 03:42:48.588515 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:42:48.588435 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc59226_544a_4420_be01_394f2654df9b.slice/crio-8751539eccf52c9426ba27097dafae70ff33da78fdbd0480722a7e0d9eeea94b WatchSource:0}: Error finding container 8751539eccf52c9426ba27097dafae70ff33da78fdbd0480722a7e0d9eeea94b: Status 404 returned error can't find the container with id 8751539eccf52c9426ba27097dafae70ff33da78fdbd0480722a7e0d9eeea94b Nov 03 03:42:49.265797 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:49.265760 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" event={"ID":"afc59226-544a-4420-be01-394f2654df9b","Type":"ContainerStarted","Data":"8751539eccf52c9426ba27097dafae70ff33da78fdbd0480722a7e0d9eeea94b"} Nov 03 03:42:50.273391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:50.273346 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" event={"ID":"afc59226-544a-4420-be01-394f2654df9b","Type":"ContainerStarted","Data":"df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5"} Nov 03 03:42:50.273819 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:50.273417 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:42:50.393717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:50.393665 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" podStartSLOduration=2.515361474 podStartE2EDuration="3.393650026s" totalImagesPullingTime="878.288552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:42:50.393290733 +0000 UTC m=+2505.423889277" watchObservedRunningTime="2026-11-03 03:42:50.393650026 +0000 UTC m=+2505.424248565" Nov 03 03:42:56.285569 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:42:56.285538 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:43:00.011268 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:43:00.011230 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:43:06.886568 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:06.886532 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth"] Nov 03 03:43:06.886971 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:06.886779 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" podUID="3360b775-e435-4aea-bd7d-ba099a17ff13" containerName="maas-api" containerID="cri-o://a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd" gracePeriod=30 Nov 03 03:43:07.164670 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.164645 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:43:07.275052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.275012 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3360b775-e435-4aea-bd7d-ba099a17ff13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3360b775-e435-4aea-bd7d-ba099a17ff13-openshift-service-ca\") pod \"3360b775-e435-4aea-bd7d-ba099a17ff13\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " Nov 03 03:43:07.275269 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.275169 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls\") pod \"3360b775-e435-4aea-bd7d-ba099a17ff13\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " Nov 03 03:43:07.275335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.275268 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-metrics-certs\") pod \"3360b775-e435-4aea-bd7d-ba099a17ff13\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " Nov 03 03:43:07.275335 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.275316 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3360b775-e435-4aea-bd7d-ba099a17ff13-kube-api-access-76qjq\" (UniqueName: \"kubernetes.io/projected/3360b775-e435-4aea-bd7d-ba099a17ff13-kube-api-access-76qjq\") pod \"3360b775-e435-4aea-bd7d-ba099a17ff13\" (UID: \"3360b775-e435-4aea-bd7d-ba099a17ff13\") " Nov 03 03:43:07.276760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.276728 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3360b775-e435-4aea-bd7d-ba099a17ff13-openshift-service-ca" pod "3360b775-e435-4aea-bd7d-ba099a17ff13" (UID: "3360b775-e435-4aea-bd7d-ba099a17ff13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:43:07.283104 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.283067 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3360b775-e435-4aea-bd7d-ba099a17ff13-kube-api-access-76qjq" pod "3360b775-e435-4aea-bd7d-ba099a17ff13" (UID: "3360b775-e435-4aea-bd7d-ba099a17ff13"). InnerVolumeSpecName "kube-api-access-76qjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:43:07.290608 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.290423 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-metrics-certs" pod "3360b775-e435-4aea-bd7d-ba099a17ff13" (UID: "3360b775-e435-4aea-bd7d-ba099a17ff13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:43:07.295186 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.294992 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls" pod "3360b775-e435-4aea-bd7d-ba099a17ff13" (UID: "3360b775-e435-4aea-bd7d-ba099a17ff13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:43:07.376772 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.376735 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:07.376772 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.376766 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3360b775-e435-4aea-bd7d-ba099a17ff13-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:07.376772 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.376777 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qjq\" (UniqueName: \"kubernetes.io/projected/3360b775-e435-4aea-bd7d-ba099a17ff13-kube-api-access-76qjq\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:07.377032 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.376789 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3360b775-e435-4aea-bd7d-ba099a17ff13-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:43:07.392432 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.392356 5117 generic.go:356] "Generic (PLEG): container finished" podID="3360b775-e435-4aea-bd7d-ba099a17ff13" containerID="a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd" exitCode=0 Nov 03 03:43:07.392583 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.392425 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" Nov 03 03:43:07.392583 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.392461 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" event={"ID":"3360b775-e435-4aea-bd7d-ba099a17ff13","Type":"ContainerDied","Data":"a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd"} Nov 03 03:43:07.392583 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.392557 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth" event={"ID":"3360b775-e435-4aea-bd7d-ba099a17ff13","Type":"ContainerDied","Data":"2aaf8e9f822df4ec72d50b5a699e1cee52fa00b93bef3ea0ff5df33959e888a4"} Nov 03 03:43:07.392689 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.392591 5117 scope.go:122] "RemoveContainer" containerID="a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd" Nov 03 03:43:07.404112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.404094 5117 scope.go:122] "RemoveContainer" containerID="a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd" Nov 03 03:43:07.404396 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:43:07.404377 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd\": container with ID starting with a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd not found: ID does not exist" containerID="a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd" Nov 03 03:43:07.404480 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.404405 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd"} err="failed to get container status \"a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd\": rpc error: code = NotFound desc = could not find container \"a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd\": container with ID starting with a60912df1c1dc3e0c5df38b86ab74d2ba4fd3f05081212b189238b375008aadd not found: ID does not exist" Nov 03 03:43:07.562229 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.562187 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth"] Nov 03 03:43:07.566423 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:07.566396 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6e985-6db896dfc6-4fzth"] Nov 03 03:43:09.089664 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:43:09.089632 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3360b775-e435-4aea-bd7d-ba099a17ff13" path="/var/lib/kubelet/pods/3360b775-e435-4aea-bd7d-ba099a17ff13/volumes" Nov 03 03:43:30.011113 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:43:30.011076 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:44:00.011859 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:44:00.011806 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:44:30.011419 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:44:30.011372 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:44:32.307563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.307527 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj"] Nov 03 03:44:32.308113 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.308098 5117 cpu_manager.go:414] "containerMap: removing container" podUID="3360b775-e435-4aea-bd7d-ba099a17ff13" containerName="maas-api" Nov 03 03:44:32.308168 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.308114 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3360b775-e435-4aea-bd7d-ba099a17ff13" containerName="maas-api" Nov 03 03:44:32.308219 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.308208 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="3360b775-e435-4aea-bd7d-ba099a17ff13" containerName="maas-api" Nov 03 03:44:32.311619 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.311603 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.367026 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.366988 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a9e5cd-serving-cert\"" Nov 03 03:44:32.370403 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.370366 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj"] Nov 03 03:44:32.439508 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.439471 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-metrics-certs\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.439752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.439533 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.439752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.439641 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qthz\" (UniqueName: \"kubernetes.io/projected/52df8cc2-4d92-450d-a6ea-5ef581d37399-kube-api-access-9qthz\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.439752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.439677 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52df8cc2-4d92-450d-a6ea-5ef581d37399-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.540970 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.540933 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-metrics-certs\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.540970 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.540978 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.541253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.541040 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qthz\" (UniqueName: \"kubernetes.io/projected/52df8cc2-4d92-450d-a6ea-5ef581d37399-kube-api-access-9qthz\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.541253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.541064 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52df8cc2-4d92-450d-a6ea-5ef581d37399-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.541253 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:44:32.541167 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-serving-cert: secret "maas-api-e2e-praxis-b-a9e5cd-serving-cert" not found Nov 03 03:44:32.541253 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:44:32.541247 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls podName:52df8cc2-4d92-450d-a6ea-5ef581d37399 nodeName:}" failed. No retries permitted until 2026-11-03 03:44:33.041231804 +0000 UTC m=+2608.071830341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls") pod "maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" (UID: "52df8cc2-4d92-450d-a6ea-5ef581d37399") : secret "maas-api-e2e-praxis-b-a9e5cd-serving-cert" not found Nov 03 03:44:32.541821 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.541803 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52df8cc2-4d92-450d-a6ea-5ef581d37399-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.543924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.543890 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-metrics-certs\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:32.604066 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:32.604029 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qthz\" (UniqueName: \"kubernetes.io/projected/52df8cc2-4d92-450d-a6ea-5ef581d37399-kube-api-access-9qthz\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:33.048075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:33.047976 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:33.050804 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:33.050777 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:33.225096 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:33.225059 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:33.608494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:33.608424 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj"] Nov 03 03:44:33.609605 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:44:33.609577 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52df8cc2_4d92_450d_a6ea_5ef581d37399.slice/crio-69b4a50a58c9dfece7d8233c0bb5383d4b7a6001ec5747703612da2403ab9156 WatchSource:0}: Error finding container 69b4a50a58c9dfece7d8233c0bb5383d4b7a6001ec5747703612da2403ab9156: Status 404 returned error can't find the container with id 69b4a50a58c9dfece7d8233c0bb5383d4b7a6001ec5747703612da2403ab9156 Nov 03 03:44:33.611040 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:33.611024 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:44:33.926438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:33.926348 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" event={"ID":"52df8cc2-4d92-450d-a6ea-5ef581d37399","Type":"ContainerStarted","Data":"69b4a50a58c9dfece7d8233c0bb5383d4b7a6001ec5747703612da2403ab9156"} Nov 03 03:44:34.934555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:34.934520 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" event={"ID":"52df8cc2-4d92-450d-a6ea-5ef581d37399","Type":"ContainerStarted","Data":"4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8"} Nov 03 03:44:34.935051 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:34.934605 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:44:35.054474 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:35.054386 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" podStartSLOduration=2.208995928 podStartE2EDuration="3.054372236s" totalImagesPullingTime="845.376308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:35.05383554 +0000 UTC m=+2610.084434079" watchObservedRunningTime="2026-11-03 03:44:35.054372236 +0000 UTC m=+2610.084970773" Nov 03 03:44:40.946776 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:44:40.946743 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:45:00.010939 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:45:00.010896 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:45:30.014806 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:45:30.014715 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:45:45.503102 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.503067 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf"] Nov 03 03:45:45.503667 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.503286 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" podUID="384db0de-a7df-47e8-8f4d-f7fdb7982434" containerName="maas-api" containerID="cri-o://748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a" gracePeriod=30 Nov 03 03:45:45.800555 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.800527 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:45:45.855011 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.854976 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls\") pod \"384db0de-a7df-47e8-8f4d-f7fdb7982434\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " Nov 03 03:45:45.855213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.855029 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384db0de-a7df-47e8-8f4d-f7fdb7982434-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384db0de-a7df-47e8-8f4d-f7fdb7982434-openshift-service-ca\") pod \"384db0de-a7df-47e8-8f4d-f7fdb7982434\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " Nov 03 03:45:45.855213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.855062 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384db0de-a7df-47e8-8f4d-f7fdb7982434-kube-api-access-jh692\" (UniqueName: \"kubernetes.io/projected/384db0de-a7df-47e8-8f4d-f7fdb7982434-kube-api-access-jh692\") pod \"384db0de-a7df-47e8-8f4d-f7fdb7982434\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " Nov 03 03:45:45.855213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.855176 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-metrics-certs\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-metrics-certs\") pod \"384db0de-a7df-47e8-8f4d-f7fdb7982434\" (UID: \"384db0de-a7df-47e8-8f4d-f7fdb7982434\") " Nov 03 03:45:45.855489 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.855461 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384db0de-a7df-47e8-8f4d-f7fdb7982434-openshift-service-ca" pod "384db0de-a7df-47e8-8f4d-f7fdb7982434" (UID: "384db0de-a7df-47e8-8f4d-f7fdb7982434"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:45.855660 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.855636 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/384db0de-a7df-47e8-8f4d-f7fdb7982434-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:45.857322 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.857297 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls" pod "384db0de-a7df-47e8-8f4d-f7fdb7982434" (UID: "384db0de-a7df-47e8-8f4d-f7fdb7982434"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:45:45.857516 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.857493 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-metrics-certs" pod "384db0de-a7df-47e8-8f4d-f7fdb7982434" (UID: "384db0de-a7df-47e8-8f4d-f7fdb7982434"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:45:45.857796 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.857773 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384db0de-a7df-47e8-8f4d-f7fdb7982434-kube-api-access-jh692" pod "384db0de-a7df-47e8-8f4d-f7fdb7982434" (UID: "384db0de-a7df-47e8-8f4d-f7fdb7982434"). InnerVolumeSpecName "kube-api-access-jh692". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:45.957121 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.957086 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:45.957121 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.957122 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/384db0de-a7df-47e8-8f4d-f7fdb7982434-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:45.957330 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:45.957140 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh692\" (UniqueName: \"kubernetes.io/projected/384db0de-a7df-47e8-8f4d-f7fdb7982434-kube-api-access-jh692\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:46.360115 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.360079 5117 generic.go:356] "Generic (PLEG): container finished" podID="384db0de-a7df-47e8-8f4d-f7fdb7982434" containerID="748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a" exitCode=0 Nov 03 03:45:46.360115 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.360119 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" event={"ID":"384db0de-a7df-47e8-8f4d-f7fdb7982434","Type":"ContainerDied","Data":"748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a"} Nov 03 03:45:46.360342 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.360141 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" Nov 03 03:45:46.360342 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.360170 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf" event={"ID":"384db0de-a7df-47e8-8f4d-f7fdb7982434","Type":"ContainerDied","Data":"0bd66732d66f228b16bd9eb7f9e8573f22575a6f208988f5f1e2f4b4d843943b"} Nov 03 03:45:46.360342 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.360191 5117 scope.go:122] "RemoveContainer" containerID="748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a" Nov 03 03:45:46.370819 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.370802 5117 scope.go:122] "RemoveContainer" containerID="748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a" Nov 03 03:45:46.371064 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:45:46.371045 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a\": container with ID starting with 748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a not found: ID does not exist" containerID="748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a" Nov 03 03:45:46.371118 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.371075 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a"} err="failed to get container status \"748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a\": rpc error: code = NotFound desc = could not find container \"748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a\": container with ID starting with 748763cb91e8c67fb3a9008f7bdf009b56208d5a6be48c0835169ed376423b4a not found: ID does not exist" Nov 03 03:45:46.527737 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.527691 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf"] Nov 03 03:45:46.531358 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:46.531336 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-754aeb-55cbd95d4c-tkbnf"] Nov 03 03:45:47.089916 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:47.089883 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384db0de-a7df-47e8-8f4d-f7fdb7982434" path="/var/lib/kubelet/pods/384db0de-a7df-47e8-8f4d-f7fdb7982434/volumes" Nov 03 03:45:51.562930 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.562894 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg"] Nov 03 03:45:51.563692 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.563142 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" podUID="970bce3e-7dd5-4039-b7d8-27e8b3b67a99" containerName="maas-api" containerID="cri-o://d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e" gracePeriod=30 Nov 03 03:45:51.669006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.668325 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22"] Nov 03 03:45:51.669006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.668630 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" containerID="cri-o://873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48" gracePeriod=30 Nov 03 03:45:51.850925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.850898 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:45:51.920967 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.920935 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-kube-api-access-rm7h5\" (UniqueName: \"kubernetes.io/projected/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-kube-api-access-rm7h5\") pod \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " Nov 03 03:45:51.921134 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.921049 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls\") pod \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " Nov 03 03:45:51.921134 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.921096 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-metrics-certs\") pod \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " Nov 03 03:45:51.921253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.921174 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-openshift-service-ca\") pod \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\" (UID: \"970bce3e-7dd5-4039-b7d8-27e8b3b67a99\") " Nov 03 03:45:51.921638 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.921602 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-openshift-service-ca" pod "970bce3e-7dd5-4039-b7d8-27e8b3b67a99" (UID: "970bce3e-7dd5-4039-b7d8-27e8b3b67a99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:51.923273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.923241 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-kube-api-access-rm7h5" pod "970bce3e-7dd5-4039-b7d8-27e8b3b67a99" (UID: "970bce3e-7dd5-4039-b7d8-27e8b3b67a99"). InnerVolumeSpecName "kube-api-access-rm7h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:51.923382 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.923363 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls" pod "970bce3e-7dd5-4039-b7d8-27e8b3b67a99" (UID: "970bce3e-7dd5-4039-b7d8-27e8b3b67a99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:45:51.923542 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:51.923522 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-metrics-certs" pod "970bce3e-7dd5-4039-b7d8-27e8b3b67a99" (UID: "970bce3e-7dd5-4039-b7d8-27e8b3b67a99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:45:52.022763 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.022729 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm7h5\" (UniqueName: \"kubernetes.io/projected/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-kube-api-access-rm7h5\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:52.022763 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.022757 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:52.022763 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.022767 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:52.022986 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.022777 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bce3e-7dd5-4039-b7d8-27e8b3b67a99-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:52.050268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.050235 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv"] Nov 03 03:45:52.050516 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.050477 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" containerID="cri-o://ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a" gracePeriod=30 Nov 03 03:45:52.398072 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.398039 5117 generic.go:356] "Generic (PLEG): container finished" podID="970bce3e-7dd5-4039-b7d8-27e8b3b67a99" containerID="d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e" exitCode=0 Nov 03 03:45:52.398256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.398090 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" Nov 03 03:45:52.398256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.398128 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" event={"ID":"970bce3e-7dd5-4039-b7d8-27e8b3b67a99","Type":"ContainerDied","Data":"d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e"} Nov 03 03:45:52.398256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.398186 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg" event={"ID":"970bce3e-7dd5-4039-b7d8-27e8b3b67a99","Type":"ContainerDied","Data":"1ede525098bc7c7718063133c67cdc15cf2089ae898cb05b7b77074cc779ba5c"} Nov 03 03:45:52.398256 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.398216 5117 scope.go:122] "RemoveContainer" containerID="d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e" Nov 03 03:45:52.410343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.410321 5117 scope.go:122] "RemoveContainer" containerID="d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e" Nov 03 03:45:52.410664 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:45:52.410632 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e\": container with ID starting with d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e not found: ID does not exist" containerID="d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e" Nov 03 03:45:52.410721 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.410666 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e"} err="failed to get container status \"d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e\": rpc error: code = NotFound desc = could not find container \"d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e\": container with ID starting with d51cfb82282e7117fab86b57708d1040b5dd12bcd0ef594c1f4a8b68a4a6e16e not found: ID does not exist" Nov 03 03:45:52.568255 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.568214 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg"] Nov 03 03:45:52.577157 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:52.577131 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a6b9ff-79967bf9b9-4mqlg"] Nov 03 03:45:53.089862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:53.089830 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970bce3e-7dd5-4039-b7d8-27e8b3b67a99" path="/var/lib/kubelet/pods/970bce3e-7dd5-4039-b7d8-27e8b3b67a99/volumes" Nov 03 03:45:56.953398 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:56.953328 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:45:57.075678 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.075644 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-token\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-token\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.075832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.075800 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b351c41d-0767-47cc-9221-fa0ea76bfe90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b351c41d-0767-47cc-9221-fa0ea76bfe90-istiod-ca-cert\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.075907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.075843 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-credential-socket\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.075907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.075891 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-kube-api-access-fk9qz\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-kube-api-access-fk9qz\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.076019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.075940 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-envoy\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.076019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.075974 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-data\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.076182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076028 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-podinfo\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.076182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076056 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-certs\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.076182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076088 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-socket\") pod \"b351c41d-0767-47cc-9221-fa0ea76bfe90\" (UID: \"b351c41d-0767-47cc-9221-fa0ea76bfe90\") " Nov 03 03:45:57.077243 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076615 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-credential-socket" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.077243 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076630 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-data" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.077243 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076849 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b351c41d-0767-47cc-9221-fa0ea76bfe90-istiod-ca-cert" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:57.077243 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076970 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-socket" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.077243 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.076899 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-certs" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.078522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.078414 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b351c41d-0767-47cc-9221-fa0ea76bfe90-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.078768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.078753 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.078900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.078881 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.079029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.078904 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.079029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.078921 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.079029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.078709 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-token" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:57.079531 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.079505 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-kube-api-access-fk9qz" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "kube-api-access-fk9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:57.091822 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.091790 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-envoy" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.102933 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.102878 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-podinfo" pod "b351c41d-0767-47cc-9221-fa0ea76bfe90" (UID: "b351c41d-0767-47cc-9221-fa0ea76bfe90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:45:57.180126 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.180076 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9qz\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-kube-api-access-fk9qz\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.180126 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.180114 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.180126 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.180129 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.180384 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.180149 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b351c41d-0767-47cc-9221-fa0ea76bfe90-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.311348 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.311322 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:45:57.432264 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.432225 5117 generic.go:356] "Generic (PLEG): container finished" podID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerID="ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a" exitCode=0 Nov 03 03:45:57.432494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.432283 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" Nov 03 03:45:57.432494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.432311 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" event={"ID":"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6","Type":"ContainerDied","Data":"ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a"} Nov 03 03:45:57.432494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.432372 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv" event={"ID":"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6","Type":"ContainerDied","Data":"7b3bd738ae0bce57eb3d6b6b4bc7308fb51d77da0f41f41ab6715ff560a3fd40"} Nov 03 03:45:57.432494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.432401 5117 scope.go:122] "RemoveContainer" containerID="ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a" Nov 03 03:45:57.434601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.434576 5117 generic.go:356] "Generic (PLEG): container finished" podID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerID="873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48" exitCode=0 Nov 03 03:45:57.434743 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.434612 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" event={"ID":"b351c41d-0767-47cc-9221-fa0ea76bfe90","Type":"ContainerDied","Data":"873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48"} Nov 03 03:45:57.434743 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.434640 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" Nov 03 03:45:57.434743 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.434654 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22" event={"ID":"b351c41d-0767-47cc-9221-fa0ea76bfe90","Type":"ContainerDied","Data":"73eff9e8442f3d98fe6c61c295e383c22a9249819cb4c04fe2f85003894f82e0"} Nov 03 03:45:57.450026 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.450003 5117 scope.go:122] "RemoveContainer" containerID="ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a" Nov 03 03:45:57.450417 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:45:57.450398 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a\": container with ID starting with ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a not found: ID does not exist" containerID="ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a" Nov 03 03:45:57.450525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.450429 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a"} err="failed to get container status \"ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a\": rpc error: code = NotFound desc = could not find container \"ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a\": container with ID starting with ce7c6f087dbe41a7874cfd7a7cab7547b9e35c4aa7921bc086460e124f85732a not found: ID does not exist" Nov 03 03:45:57.450525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.450475 5117 scope.go:122] "RemoveContainer" containerID="873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48" Nov 03 03:45:57.462718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.462701 5117 scope.go:122] "RemoveContainer" containerID="873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48" Nov 03 03:45:57.463032 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:45:57.463009 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48\": container with ID starting with 873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48 not found: ID does not exist" containerID="873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48" Nov 03 03:45:57.463130 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.463039 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48"} err="failed to get container status \"873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48\": rpc error: code = NotFound desc = could not find container \"873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48\": container with ID starting with 873e7c3e8ea24e5a33a8c99330ff7f1053f5fd7d0724180226c85f7a64c2df48 not found: ID does not exist" Nov 03 03:45:57.483401 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483379 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-socket\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483410 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-token\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-token\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483517 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-data\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483548 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-credential-socket\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483702 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483593 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istiod-ca-cert\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483702 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483664 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-kube-api-access-5bhzg\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-kube-api-access-5bhzg\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483795 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483721 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-socket" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.483795 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483750 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-certs\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483795 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483794 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-envoy\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483836 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-podinfo\") pod \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\" (UID: \"6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6\") " Nov 03 03:45:57.483990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483836 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-data" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.483990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.483941 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-credential-socket" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.484146 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.484012 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-certs" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.484146 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.484046 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istiod-ca-cert" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:45:57.484319 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.484295 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.484387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.484325 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.484387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.484340 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.484387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.484353 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.484387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.484367 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.486194 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.486162 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-envoy" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:45:57.486529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.486509 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-podinfo" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:45:57.486639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.486566 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-token" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:57.486639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.486579 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-kube-api-access-5bhzg" pod "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" (UID: "6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6"). InnerVolumeSpecName "kube-api-access-5bhzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:45:57.585116 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.585084 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.585287 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.585122 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhzg\" (UniqueName: \"kubernetes.io/projected/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-kube-api-access-5bhzg\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.585287 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.585139 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.585287 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.585153 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:45:57.601363 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.601328 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22"] Nov 03 03:45:57.608546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.608520 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5qx22"] Nov 03 03:45:57.855216 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.855133 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv"] Nov 03 03:45:57.859205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:57.859181 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-ghbgv"] Nov 03 03:45:58.848096 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:58.848059 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t"] Nov 03 03:45:58.848628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:58.848403 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" containerID="cri-o://1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733" gracePeriod=30 Nov 03 03:45:59.090279 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:59.090247 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" path="/var/lib/kubelet/pods/6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6/volumes" Nov 03 03:45:59.090720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:59.090706 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" path="/var/lib/kubelet/pods/b351c41d-0767-47cc-9221-fa0ea76bfe90/volumes" Nov 03 03:45:59.500080 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:59.499999 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:45:59.500245 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:45:59.500076 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:46:00.015051 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:46:00.014945 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:46:04.133057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.133000 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:46:04.258893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.258864 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-kube-api-access-zjq9g\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-kube-api-access-zjq9g\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259091 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.258933 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-data\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259091 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.258971 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857e8c66-4d5f-4757-b916-056da42d0e3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857e8c66-4d5f-4757-b916-056da42d0e3c-istiod-ca-cert\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259091 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259003 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-podinfo\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259091 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259040 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-envoy\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259097 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-socket\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259152 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-certs\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259172 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-credential-socket\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259229 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-token\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-token\") pod \"857e8c66-4d5f-4757-b916-056da42d0e3c\" (UID: \"857e8c66-4d5f-4757-b916-056da42d0e3c\") " Nov 03 03:46:04.259310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259289 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-data" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:04.259588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259437 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-socket" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:04.259588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259541 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-credential-socket" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:04.259588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259540 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857e8c66-4d5f-4757-b916-056da42d0e3c-istiod-ca-cert" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:04.259721 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259700 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.259798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259728 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.259798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259745 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.259798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259760 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/857e8c66-4d5f-4757-b916-056da42d0e3c-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.259959 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.259823 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-certs" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:04.262203 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.262177 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-envoy" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:46:04.262413 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.262387 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-token" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:04.262545 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.262523 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-podinfo" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:46:04.262668 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.262647 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-kube-api-access-zjq9g" pod "857e8c66-4d5f-4757-b916-056da42d0e3c" (UID: "857e8c66-4d5f-4757-b916-056da42d0e3c"). InnerVolumeSpecName "kube-api-access-zjq9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:04.360313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.360282 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.360313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.360311 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjq9g\" (UniqueName: \"kubernetes.io/projected/857e8c66-4d5f-4757-b916-056da42d0e3c-kube-api-access-zjq9g\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.360530 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.360322 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.360530 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.360333 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.360530 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.360343 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/857e8c66-4d5f-4757-b916-056da42d0e3c-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:04.486169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.486079 5117 generic.go:356] "Generic (PLEG): container finished" podID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerID="1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733" exitCode=0 Nov 03 03:46:04.486169 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.486141 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" Nov 03 03:46:04.486368 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.486168 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" event={"ID":"857e8c66-4d5f-4757-b916-056da42d0e3c","Type":"ContainerDied","Data":"1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733"} Nov 03 03:46:04.486368 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.486232 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t" event={"ID":"857e8c66-4d5f-4757-b916-056da42d0e3c","Type":"ContainerDied","Data":"1a1ffd4c2f7a0207b31e2bd079ba2ec7922409967e39f2f0e727ef2be035f11e"} Nov 03 03:46:04.486368 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.486247 5117 scope.go:122] "RemoveContainer" containerID="1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733" Nov 03 03:46:04.497142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.497126 5117 scope.go:122] "RemoveContainer" containerID="1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733" Nov 03 03:46:04.497495 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:46:04.497414 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733\": container with ID starting with 1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733 not found: ID does not exist" containerID="1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733" Nov 03 03:46:04.497495 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.497460 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733"} err="failed to get container status \"1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733\": rpc error: code = NotFound desc = could not find container \"1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733\": container with ID starting with 1db910321461004d4a6e3fe9df8a4c5e1aa93530dc2d3711f78e112b29ad1733 not found: ID does not exist" Nov 03 03:46:04.655589 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.655535 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t"] Nov 03 03:46:04.660798 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:04.660771 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-c8p8t"] Nov 03 03:46:05.089997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:05.089961 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" path="/var/lib/kubelet/pods/857e8c66-4d5f-4757-b916-056da42d0e3c/volumes" Nov 03 03:46:07.520651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:07.520624 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:46:07.527917 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:07.527888 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:46:30.010302 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:46:30.010260 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:46:41.458200 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.458122 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w"] Nov 03 03:46:41.458631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.458344 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" podUID="56bb8917-cc8f-42ce-a4de-e4e92bccc348" containerName="maas-api" containerID="cri-o://3002a88a11cefbd44d7ea7ee163e7a09ed2a67d44ffb931826be65b67eca30fe" gracePeriod=30 Nov 03 03:46:41.708955 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.708871 5117 generic.go:356] "Generic (PLEG): container finished" podID="56bb8917-cc8f-42ce-a4de-e4e92bccc348" containerID="3002a88a11cefbd44d7ea7ee163e7a09ed2a67d44ffb931826be65b67eca30fe" exitCode=0 Nov 03 03:46:41.709122 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.708950 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" event={"ID":"56bb8917-cc8f-42ce-a4de-e4e92bccc348","Type":"ContainerDied","Data":"3002a88a11cefbd44d7ea7ee163e7a09ed2a67d44ffb931826be65b67eca30fe"} Nov 03 03:46:41.735811 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.735782 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:46:41.850499 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.850462 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bb8917-cc8f-42ce-a4de-e4e92bccc348-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56bb8917-cc8f-42ce-a4de-e4e92bccc348-openshift-service-ca\") pod \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " Nov 03 03:46:41.850712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.850556 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bb8917-cc8f-42ce-a4de-e4e92bccc348-kube-api-access-cmfcs\" (UniqueName: \"kubernetes.io/projected/56bb8917-cc8f-42ce-a4de-e4e92bccc348-kube-api-access-cmfcs\") pod \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " Nov 03 03:46:41.850712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.850663 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-metrics-certs\") pod \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " Nov 03 03:46:41.850839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.850765 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls\") pod \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\" (UID: \"56bb8917-cc8f-42ce-a4de-e4e92bccc348\") " Nov 03 03:46:41.850839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.850777 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bb8917-cc8f-42ce-a4de-e4e92bccc348-openshift-service-ca" pod "56bb8917-cc8f-42ce-a4de-e4e92bccc348" (UID: "56bb8917-cc8f-42ce-a4de-e4e92bccc348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:46:41.851124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.851106 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56bb8917-cc8f-42ce-a4de-e4e92bccc348-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:41.853590 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.853558 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls" pod "56bb8917-cc8f-42ce-a4de-e4e92bccc348" (UID: "56bb8917-cc8f-42ce-a4de-e4e92bccc348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:46:41.853590 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.853579 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bb8917-cc8f-42ce-a4de-e4e92bccc348-kube-api-access-cmfcs" pod "56bb8917-cc8f-42ce-a4de-e4e92bccc348" (UID: "56bb8917-cc8f-42ce-a4de-e4e92bccc348"). InnerVolumeSpecName "kube-api-access-cmfcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:46:41.853898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.853880 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-metrics-certs" pod "56bb8917-cc8f-42ce-a4de-e4e92bccc348" (UID: "56bb8917-cc8f-42ce-a4de-e4e92bccc348"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:46:41.951853 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.951822 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:41.951853 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.951848 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56bb8917-cc8f-42ce-a4de-e4e92bccc348-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:41.951853 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:41.951859 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfcs\" (UniqueName: \"kubernetes.io/projected/56bb8917-cc8f-42ce-a4de-e4e92bccc348-kube-api-access-cmfcs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:46:42.716640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:42.716606 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" Nov 03 03:46:42.716640 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:42.716616 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w" event={"ID":"56bb8917-cc8f-42ce-a4de-e4e92bccc348","Type":"ContainerDied","Data":"5b5e66a03b02f1d42a19f9cf6cab838fae26acddbdb55bcd399a710bcf8f596d"} Nov 03 03:46:42.717184 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:42.716696 5117 scope.go:122] "RemoveContainer" containerID="3002a88a11cefbd44d7ea7ee163e7a09ed2a67d44ffb931826be65b67eca30fe" Nov 03 03:46:42.884313 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:42.884275 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w"] Nov 03 03:46:42.934890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:42.934855 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b755ee-7fb6b86f6-25v5w"] Nov 03 03:46:43.089924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:43.089895 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bb8917-cc8f-42ce-a4de-e4e92bccc348" path="/var/lib/kubelet/pods/56bb8917-cc8f-42ce-a4de-e4e92bccc348/volumes" Nov 03 03:46:46.716210 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716175 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls"] Nov 03 03:46:46.716748 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716729 5117 cpu_manager.go:414] "containerMap: removing container" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" Nov 03 03:46:46.716832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716750 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" Nov 03 03:46:46.716832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716774 5117 cpu_manager.go:414] "containerMap: removing container" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" Nov 03 03:46:46.716832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716783 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" Nov 03 03:46:46.716832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716796 5117 cpu_manager.go:414] "containerMap: removing container" podUID="970bce3e-7dd5-4039-b7d8-27e8b3b67a99" containerName="maas-api" Nov 03 03:46:46.716832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716803 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970bce3e-7dd5-4039-b7d8-27e8b3b67a99" containerName="maas-api" Nov 03 03:46:46.716832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716822 5117 cpu_manager.go:414] "containerMap: removing container" podUID="56bb8917-cc8f-42ce-a4de-e4e92bccc348" containerName="maas-api" Nov 03 03:46:46.716832 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716829 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bb8917-cc8f-42ce-a4de-e4e92bccc348" containerName="maas-api" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716857 5117 cpu_manager.go:414] "containerMap: removing container" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716867 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716885 5117 cpu_manager.go:414] "containerMap: removing container" podUID="384db0de-a7df-47e8-8f4d-f7fdb7982434" containerName="maas-api" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716892 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384db0de-a7df-47e8-8f4d-f7fdb7982434" containerName="maas-api" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.716998 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="b351c41d-0767-47cc-9221-fa0ea76bfe90" containerName="istio-proxy" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.717009 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="384db0de-a7df-47e8-8f4d-f7fdb7982434" containerName="maas-api" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.717018 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f571655-0c7e-44a5-a8e4-fa4ef0ac09f6" containerName="istio-proxy" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.717024 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="857e8c66-4d5f-4757-b916-056da42d0e3c" containerName="istio-proxy" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.717033 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="970bce3e-7dd5-4039-b7d8-27e8b3b67a99" containerName="maas-api" Nov 03 03:46:46.717103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.717044 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bb8917-cc8f-42ce-a4de-e4e92bccc348" containerName="maas-api" Nov 03 03:46:46.721679 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.721660 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.775075 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.775029 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-29dd3f\"/\"openshift-service-ca.crt\"" Nov 03 03:46:46.797341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.797297 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-dshm\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.797539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.797367 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-home\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.797539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.797492 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tls-certs\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.797654 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.797616 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d9m\" (UniqueName: \"kubernetes.io/projected/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-kube-api-access-k6d9m\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.797752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.797728 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-model-cache\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.797823 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.797761 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tmp-dir\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.828125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.825568 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-29dd3f\"/\"default-dockercfg-6mknb\"" Nov 03 03:46:46.828125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.825580 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-29dd3f\"/\"kube-root-ca.crt\"" Nov 03 03:46:46.828125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.825638 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-29dd3f\"/\"praxis-route-29dd3f-kserve-self-signed-certs\"" Nov 03 03:46:46.829054 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.829027 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls"] Nov 03 03:46:46.898870 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.898830 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-model-cache\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.898870 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.898870 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tmp-dir\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.899073 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.898997 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-dshm\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.899110 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.899084 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-home\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.899149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.899119 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tls-certs\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.899201 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.899160 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d9m\" (UniqueName: \"kubernetes.io/projected/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-kube-api-access-k6d9m\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.899253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.899231 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tmp-dir\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.899317 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.899288 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-model-cache\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.899476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.899402 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-home\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.901553 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.901534 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-dshm\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.901708 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.901690 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tls-certs\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:46.963077 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:46.963047 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d9m\" (UniqueName: \"kubernetes.io/projected/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-kube-api-access-k6d9m\") pod \"praxis-route-29dd3f-kserve-6569c5dc89-j6rls\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:47.034367 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:47.034280 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:47.425790 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:47.425755 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls"] Nov 03 03:46:47.426066 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:46:47.426040 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod644f8c0b_ed4b_4f73_8e01_d4f9ff511a5a.slice/crio-2784f2d60dfdcc62288afa20ece82062ebfba9e0fa498198282fbe922469c935 WatchSource:0}: Error finding container 2784f2d60dfdcc62288afa20ece82062ebfba9e0fa498198282fbe922469c935: Status 404 returned error can't find the container with id 2784f2d60dfdcc62288afa20ece82062ebfba9e0fa498198282fbe922469c935 Nov 03 03:46:47.747072 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:47.746979 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" event={"ID":"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a","Type":"ContainerStarted","Data":"f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05"} Nov 03 03:46:47.747072 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:47.747027 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" event={"ID":"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a","Type":"ContainerStarted","Data":"2784f2d60dfdcc62288afa20ece82062ebfba9e0fa498198282fbe922469c935"} Nov 03 03:46:47.870290 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:47.870236 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" podStartSLOduration=1.870224203 podStartE2EDuration="1.870224203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:46:47.868949634 +0000 UTC m=+2742.899548172" watchObservedRunningTime="2026-11-03 03:46:47.870224203 +0000 UTC m=+2742.900822740" Nov 03 03:46:48.035477 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:48.035378 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:57.034584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:57.034540 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:57.047761 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:57.047723 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:46:57.822780 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:46:57.822749 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:47:00.011868 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:47:00.011830 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:47:10.760809 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.760774 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8"] Nov 03 03:47:10.765970 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.765950 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.852293 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.852259 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-metrics-certs\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.852462 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.852305 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d205e-f332-4353-ba6c-e79e24aede56-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.852462 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.852328 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.852566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.852506 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wvg\" (UniqueName: \"kubernetes.io/projected/9a4d205e-f332-4353-ba6c-e79e24aede56-kube-api-access-97wvg\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.871004 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.870977 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5ec4ce-serving-cert\"" Nov 03 03:47:10.873521 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.873499 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8"] Nov 03 03:47:10.953303 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.953267 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-metrics-certs\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.953303 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.953306 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d205e-f332-4353-ba6c-e79e24aede56-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.953560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.953329 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.953560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.953424 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wvg\" (UniqueName: \"kubernetes.io/projected/9a4d205e-f332-4353-ba6c-e79e24aede56-kube-api-access-97wvg\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.953560 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:47:10.953513 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-serving-cert: secret "maas-api-e2e-praxis-a-5ec4ce-serving-cert" not found Nov 03 03:47:10.953720 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:47:10.953586 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls podName:9a4d205e-f332-4353-ba6c-e79e24aede56 nodeName:}" failed. No retries permitted until 2026-11-03 03:47:11.453566338 +0000 UTC m=+2766.484164868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls") pod "maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" (UID: "9a4d205e-f332-4353-ba6c-e79e24aede56") : secret "maas-api-e2e-praxis-a-5ec4ce-serving-cert" not found Nov 03 03:47:10.953974 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.953951 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d205e-f332-4353-ba6c-e79e24aede56-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:10.956024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:10.955996 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-metrics-certs\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:11.015803 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:11.015721 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wvg\" (UniqueName: \"kubernetes.io/projected/9a4d205e-f332-4353-ba6c-e79e24aede56-kube-api-access-97wvg\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:11.457594 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:11.457560 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:11.460309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:11.460286 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:11.679462 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:11.679399 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:12.055268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:12.055233 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8"] Nov 03 03:47:12.056407 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:47:12.056363 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4d205e_f332_4353_ba6c_e79e24aede56.slice/crio-29d1ea99cae1fa1386ec8c982dd6825b0e61ba002ea00233ace9866c89e9c3cb WatchSource:0}: Error finding container 29d1ea99cae1fa1386ec8c982dd6825b0e61ba002ea00233ace9866c89e9c3cb: Status 404 returned error can't find the container with id 29d1ea99cae1fa1386ec8c982dd6825b0e61ba002ea00233ace9866c89e9c3cb Nov 03 03:47:12.909306 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:12.909271 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" event={"ID":"9a4d205e-f332-4353-ba6c-e79e24aede56","Type":"ContainerStarted","Data":"29d1ea99cae1fa1386ec8c982dd6825b0e61ba002ea00233ace9866c89e9c3cb"} Nov 03 03:47:13.916659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:13.916620 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" event={"ID":"9a4d205e-f332-4353-ba6c-e79e24aede56","Type":"ContainerStarted","Data":"cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d"} Nov 03 03:47:13.917180 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:13.916711 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:14.039233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:14.039175 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" podStartSLOduration=3.080147422 podStartE2EDuration="4.039158183s" totalImagesPullingTime="959.010761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:47:14.037016437 +0000 UTC m=+2769.067614975" watchObservedRunningTime="2026-11-03 03:47:14.039158183 +0000 UTC m=+2769.069756720" Nov 03 03:47:19.928141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:19.928105 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:47:30.011290 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:47:30.011256 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:47:50.106042 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.105997 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj"] Nov 03 03:47:50.106687 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.106262 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" podUID="52df8cc2-4d92-450d-a6ea-5ef581d37399" containerName="maas-api" containerID="cri-o://4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8" gracePeriod=30 Nov 03 03:47:50.391957 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.391919 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:47:50.557809 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.557775 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52df8cc2-4d92-450d-a6ea-5ef581d37399-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52df8cc2-4d92-450d-a6ea-5ef581d37399-openshift-service-ca\") pod \"52df8cc2-4d92-450d-a6ea-5ef581d37399\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " Nov 03 03:47:50.557965 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.557846 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52df8cc2-4d92-450d-a6ea-5ef581d37399-kube-api-access-9qthz\" (UniqueName: \"kubernetes.io/projected/52df8cc2-4d92-450d-a6ea-5ef581d37399-kube-api-access-9qthz\") pod \"52df8cc2-4d92-450d-a6ea-5ef581d37399\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " Nov 03 03:47:50.557965 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.557872 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-metrics-certs\") pod \"52df8cc2-4d92-450d-a6ea-5ef581d37399\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " Nov 03 03:47:50.558059 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.557969 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls\") pod \"52df8cc2-4d92-450d-a6ea-5ef581d37399\" (UID: \"52df8cc2-4d92-450d-a6ea-5ef581d37399\") " Nov 03 03:47:50.558221 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.558194 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52df8cc2-4d92-450d-a6ea-5ef581d37399-openshift-service-ca" pod "52df8cc2-4d92-450d-a6ea-5ef581d37399" (UID: "52df8cc2-4d92-450d-a6ea-5ef581d37399"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:47:50.558415 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.558391 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52df8cc2-4d92-450d-a6ea-5ef581d37399-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:50.560185 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.560154 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls" pod "52df8cc2-4d92-450d-a6ea-5ef581d37399" (UID: "52df8cc2-4d92-450d-a6ea-5ef581d37399"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:47:50.560185 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.560172 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-metrics-certs" pod "52df8cc2-4d92-450d-a6ea-5ef581d37399" (UID: "52df8cc2-4d92-450d-a6ea-5ef581d37399"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:47:50.560319 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.560186 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52df8cc2-4d92-450d-a6ea-5ef581d37399-kube-api-access-9qthz" pod "52df8cc2-4d92-450d-a6ea-5ef581d37399" (UID: "52df8cc2-4d92-450d-a6ea-5ef581d37399"). InnerVolumeSpecName "kube-api-access-9qthz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:47:50.660017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.659934 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:50.660017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.659965 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qthz\" (UniqueName: \"kubernetes.io/projected/52df8cc2-4d92-450d-a6ea-5ef581d37399-kube-api-access-9qthz\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:50.660017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:50.659977 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52df8cc2-4d92-450d-a6ea-5ef581d37399-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:47:51.137519 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.137477 5117 generic.go:356] "Generic (PLEG): container finished" podID="52df8cc2-4d92-450d-a6ea-5ef581d37399" containerID="4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8" exitCode=0 Nov 03 03:47:51.137990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.137540 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" Nov 03 03:47:51.137990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.137571 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" event={"ID":"52df8cc2-4d92-450d-a6ea-5ef581d37399","Type":"ContainerDied","Data":"4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8"} Nov 03 03:47:51.137990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.137634 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj" event={"ID":"52df8cc2-4d92-450d-a6ea-5ef581d37399","Type":"ContainerDied","Data":"69b4a50a58c9dfece7d8233c0bb5383d4b7a6001ec5747703612da2403ab9156"} Nov 03 03:47:51.137990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.137661 5117 scope.go:122] "RemoveContainer" containerID="4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8" Nov 03 03:47:51.147725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.147701 5117 scope.go:122] "RemoveContainer" containerID="4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8" Nov 03 03:47:51.148023 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:47:51.148003 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8\": container with ID starting with 4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8 not found: ID does not exist" containerID="4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8" Nov 03 03:47:51.148087 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.148034 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8"} err="failed to get container status \"4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8\": rpc error: code = NotFound desc = could not find container \"4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8\": container with ID starting with 4b2e3dc150d4d3d8a61423ceedea39fd882563cf8cc7857003c2aa43c9b766e8 not found: ID does not exist" Nov 03 03:47:51.305656 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.305616 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj"] Nov 03 03:47:51.356657 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:51.356577 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a9e5cd-8474f59dff-jc4pj"] Nov 03 03:47:53.089842 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:47:53.089805 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52df8cc2-4d92-450d-a6ea-5ef581d37399" path="/var/lib/kubelet/pods/52df8cc2-4d92-450d-a6ea-5ef581d37399/volumes" Nov 03 03:48:00.010634 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:48:00.010584 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:48:30.010800 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:48:30.010760 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:48:52.960893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:52.960854 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g"] Nov 03 03:48:52.961326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:52.961295 5117 cpu_manager.go:414] "containerMap: removing container" podUID="52df8cc2-4d92-450d-a6ea-5ef581d37399" containerName="maas-api" Nov 03 03:48:52.961326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:52.961305 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52df8cc2-4d92-450d-a6ea-5ef581d37399" containerName="maas-api" Nov 03 03:48:52.961422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:52.961399 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="52df8cc2-4d92-450d-a6ea-5ef581d37399" containerName="maas-api" Nov 03 03:48:52.964814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:52.964794 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.018199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.018172 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b1ad4e-openshift-default-dockercfg-m64tn\"" Nov 03 03:48:53.072045 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.072012 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g"] Nov 03 03:48:53.103867 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.103838 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.103867 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.103871 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.104057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.103898 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.104057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.104020 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.104128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.104082 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fhf\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-kube-api-access-m9fhf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.104128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.104122 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.104205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.104145 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.104205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.104191 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.104285 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.104272 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.205671 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.205637 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fhf\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-kube-api-access-m9fhf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.205840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.205682 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.205840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.205702 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.205840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.205724 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.205913 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.205989 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206016 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206064 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206117 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206129 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206237 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206395 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206362 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206471 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.206709 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.206649 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.208155 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.208126 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.208569 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.208552 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.267840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.267762 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.267961 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.267885 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fhf\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-kube-api-access-m9fhf\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.277563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.277544 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:53.703799 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.703772 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g"] Nov 03 03:48:53.704545 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:48:53.704514 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d8f94f_9b7f_450e_8bbb_b42776fd7f94.slice/crio-afdc0117a5450c7fce814c6fc1affe3eeb28a9f7dcd0f451c82f41ef08657efd WatchSource:0}: Error finding container afdc0117a5450c7fce814c6fc1affe3eeb28a9f7dcd0f451c82f41ef08657efd: Status 404 returned error can't find the container with id afdc0117a5450c7fce814c6fc1affe3eeb28a9f7dcd0f451c82f41ef08657efd Nov 03 03:48:53.706922 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.706891 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:48:53.706985 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.706957 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:48:53.707029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:53.706986 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:48:54.529342 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:54.529299 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" event={"ID":"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94","Type":"ContainerStarted","Data":"3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697"} Nov 03 03:48:54.529883 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:54.529365 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" event={"ID":"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94","Type":"ContainerStarted","Data":"afdc0117a5450c7fce814c6fc1affe3eeb28a9f7dcd0f451c82f41ef08657efd"} Nov 03 03:48:54.729999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:54.729943 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podStartSLOduration=2.729928138 podStartE2EDuration="2.729928138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:54.727874963 +0000 UTC m=+2869.758473502" watchObservedRunningTime="2026-11-03 03:48:54.729928138 +0000 UTC m=+2869.760526675" Nov 03 03:48:55.278297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:55.278266 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:48:55.279739 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:55.279715 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:48:55.279873 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:55.279772 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 03:48:56.278659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:56.278614 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:48:56.279143 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:56.278686 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 03:48:57.278241 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:57.278206 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:48:57.278421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:57.278267 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 03:48:58.278838 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:58.278794 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:48:58.279259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:58.278859 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 03:48:59.278830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:59.278795 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:48:59.279050 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:48:59.278885 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 03:49:00.014064 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:49:00.014013 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:49:00.278898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:00.278809 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 03:49:00.278898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:00.278875 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 03:49:02.318663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:02.318614 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:49:02.319379 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:02.318706 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:49:02.345088 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:02.345058 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:49:02.345584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:02.345562 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:49:02.346326 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:02.346308 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 03:49:15.450387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.450338 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg"] Nov 03 03:49:15.454539 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.454515 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.508790 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.508749 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de27f5-openshift-default-dockercfg-djfq7\"" Nov 03 03:49:15.542552 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542515 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.542727 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542628 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.542727 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542657 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.542809 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542732 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djrjv\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-kube-api-access-djrjv\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.542809 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542793 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.542872 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542836 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.542907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542885 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.542940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.542914 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.543041 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.543016 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.563833 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.563796 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg"] Nov 03 03:49:15.644260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644222 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644262 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644310 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djrjv\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-kube-api-access-djrjv\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644365 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644421 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644530 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644575 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644626 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644673 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.644931 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644908 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.645003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.644949 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.645060 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.645035 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.645116 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.645073 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.645281 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.645259 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.647242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.647200 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.647242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.647226 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.706380 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.706305 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.706541 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.706505 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djrjv\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-kube-api-access-djrjv\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:15.768120 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:15.768084 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:16.200563 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.200528 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg"] Nov 03 03:49:16.200939 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:49:16.200914 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cad53d_ab1f_47c3_91e4_97d8e432f889.slice/crio-babe898182e7671f42c63e52414b5980f13482acb460bb65b6df904eb584adb4 WatchSource:0}: Error finding container babe898182e7671f42c63e52414b5980f13482acb460bb65b6df904eb584adb4: Status 404 returned error can't find the container with id babe898182e7671f42c63e52414b5980f13482acb460bb65b6df904eb584adb4 Nov 03 03:49:16.203243 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.203211 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:49:16.203325 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.203269 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:49:16.203325 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.203295 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:49:16.675406 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.675362 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" event={"ID":"b0cad53d-ab1f-47c3-91e4-97d8e432f889","Type":"ContainerStarted","Data":"949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21"} Nov 03 03:49:16.675800 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.675424 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" event={"ID":"b0cad53d-ab1f-47c3-91e4-97d8e432f889","Type":"ContainerStarted","Data":"babe898182e7671f42c63e52414b5980f13482acb460bb65b6df904eb584adb4"} Nov 03 03:49:16.768213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.768181 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:16.799947 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:16.799878 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podStartSLOduration=1.79985726 podStartE2EDuration="1.79985726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:16.797961827 +0000 UTC m=+2891.828560366" watchObservedRunningTime="2026-11-03 03:49:16.79985726 +0000 UTC m=+2891.830455802" Nov 03 03:49:17.770272 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:17.770234 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:49:17.770703 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:17.770295 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 03:49:18.768724 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:18.768682 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:49:18.768920 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:18.768739 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 03:49:19.769022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:19.768981 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:49:19.769494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:19.769059 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 03:49:20.727810 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:20.727767 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls"] Nov 03 03:49:20.769191 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:20.769113 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:49:20.769191 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:20.769170 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 03:49:21.769054 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:21.769017 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:49:21.769210 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:21.769067 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 03:49:22.768778 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:22.768746 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 03:49:22.768942 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:22.768804 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 03:49:23.774413 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:23.774383 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:23.774910 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:23.774820 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:23.775655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:23.775635 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 03:49:30.012102 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:49:30.012051 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:49:35.749812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:35.749752 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" podUID="644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" containerName="main" containerID="cri-o://f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05" gracePeriod=45 Nov 03 03:49:35.812211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:35.812172 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" podUID="644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": dial tcp 10.132.0.103:8000: connect: connection refused" Nov 03 03:49:36.013162 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.013106 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:49:36.092152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092108 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-dshm\") pod \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " Nov 03 03:49:36.092302 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092186 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tls-certs\") pod \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " Nov 03 03:49:36.092302 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092261 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-kube-api-access-k6d9m\" (UniqueName: \"kubernetes.io/projected/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-kube-api-access-k6d9m\") pod \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " Nov 03 03:49:36.092394 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092316 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tmp-dir\") pod \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " Nov 03 03:49:36.092497 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092434 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-home\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-home\") pod \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " Nov 03 03:49:36.092564 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092519 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-model-cache\") pod \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\" (UID: \"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a\") " Nov 03 03:49:36.092623 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092591 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tmp-dir" pod "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" (UID: "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:36.092771 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092744 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-home" pod "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" (UID: "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:36.092898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092849 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-model-cache" pod "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" (UID: "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:36.093016 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.092995 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:36.093081 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.093022 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:36.093081 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.093036 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:36.094622 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.094593 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tls-certs" pod "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" (UID: "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:49:36.094972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.094947 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-dshm" pod "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" (UID: "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:49:36.094972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.094967 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-kube-api-access-k6d9m" pod "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" (UID: "644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a"). InnerVolumeSpecName "kube-api-access-k6d9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:49:36.194197 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.194162 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6d9m\" (UniqueName: \"kubernetes.io/projected/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-kube-api-access-k6d9m\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:36.194197 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.194189 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:36.194197 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.194198 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:49:36.813862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.813826 5117 generic.go:356] "Generic (PLEG): container finished" podID="644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" containerID="f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05" exitCode=0 Nov 03 03:49:36.814343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.813887 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" Nov 03 03:49:36.814343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.813919 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" event={"ID":"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a","Type":"ContainerDied","Data":"f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05"} Nov 03 03:49:36.814343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.813960 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls" event={"ID":"644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a","Type":"ContainerDied","Data":"2784f2d60dfdcc62288afa20ece82062ebfba9e0fa498198282fbe922469c935"} Nov 03 03:49:36.814343 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.813982 5117 scope.go:122] "RemoveContainer" containerID="f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05" Nov 03 03:49:36.827912 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.827891 5117 scope.go:122] "RemoveContainer" containerID="f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05" Nov 03 03:49:36.828301 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:49:36.828282 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05\": container with ID starting with f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05 not found: ID does not exist" containerID="f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05" Nov 03 03:49:36.828387 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.828310 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05"} err="failed to get container status \"f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05\": rpc error: code = NotFound desc = could not find container \"f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05\": container with ID starting with f6f5bc2e770215a38ece4d605f7c0faa9c00ec0d5878f97f8d8d34d021663c05 not found: ID does not exist" Nov 03 03:49:36.983495 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:36.983433 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls"] Nov 03 03:49:37.034265 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:37.034232 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-29dd3f/praxis-route-29dd3f-kserve-6569c5dc89-j6rls"] Nov 03 03:49:37.089664 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:37.089593 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" path="/var/lib/kubelet/pods/644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a/volumes" Nov 03 03:49:52.954320 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.954287 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt"] Nov 03 03:49:52.954852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.954837 5117 cpu_manager.go:414] "containerMap: removing container" podUID="644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" containerName="main" Nov 03 03:49:52.954911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.954853 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" containerName="main" Nov 03 03:49:52.954948 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.954929 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="644f8c0b-ed4b-4f73-8e01-d4f9ff511a5a" containerName="main" Nov 03 03:49:52.958486 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.958465 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.972789 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.972764 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.972893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.972827 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.972934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.972916 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.972970 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.972955 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.973029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.973014 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.973067 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.973036 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.973113 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.973101 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.973148 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.973134 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15074a77-6cf3-4d5b-b9f5-e275361bee20-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:52.973182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:52.973169 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwcr\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-kube-api-access-4hwcr\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.065815 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.065786 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt"] Nov 03 03:49:53.073887 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.073860 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.073917 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15074a77-6cf3-4d5b-b9f5-e275361bee20-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.073974 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwcr\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-kube-api-access-4hwcr\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074016 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074100 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074149 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074196 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074240 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074424 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074263 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074597 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074572 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074680 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074595 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074813 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074787 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074808 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15074a77-6cf3-4d5b-b9f5-e275361bee20-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.074890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.074853 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.076662 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.076640 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.077003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.076981 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.136412 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.136382 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.136578 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.136425 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwcr\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-kube-api-access-4hwcr\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.271332 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.271255 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:53.700398 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.700369 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt"] Nov 03 03:49:53.701585 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:49:53.701555 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15074a77_6cf3_4d5b_b9f5_e275361bee20.slice/crio-5e57ffe0d167726d31c8067907f57ea17f4eaa438fead995774b051d11650663 WatchSource:0}: Error finding container 5e57ffe0d167726d31c8067907f57ea17f4eaa438fead995774b051d11650663: Status 404 returned error can't find the container with id 5e57ffe0d167726d31c8067907f57ea17f4eaa438fead995774b051d11650663 Nov 03 03:49:53.703972 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.703933 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:49:53.704085 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.704017 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:49:53.704085 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.704063 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:49:53.928973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.928935 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" event={"ID":"15074a77-6cf3-4d5b-b9f5-e275361bee20","Type":"ContainerStarted","Data":"3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868"} Nov 03 03:49:53.929134 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:53.928996 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" event={"ID":"15074a77-6cf3-4d5b-b9f5-e275361bee20","Type":"ContainerStarted","Data":"5e57ffe0d167726d31c8067907f57ea17f4eaa438fead995774b051d11650663"} Nov 03 03:49:54.054233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:54.054114 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podStartSLOduration=2.054094822 podStartE2EDuration="2.054094822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:54.051956957 +0000 UTC m=+2929.082555492" watchObservedRunningTime="2026-11-03 03:49:54.054094822 +0000 UTC m=+2929.084693361" Nov 03 03:49:54.272189 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:54.272161 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:49:54.273513 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:54.273491 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 03:49:54.273635 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:54.273544 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 03:49:55.271721 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:55.271690 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 03:49:55.272103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:55.271743 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 03:49:56.272062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:56.272027 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 03:49:56.272601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:56.272081 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 03:49:57.272111 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:57.272064 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 03:49:57.272647 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:57.272140 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 03:49:58.271562 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:58.271526 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 03:49:58.271764 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:58.271581 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 03:49:59.274786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:59.274749 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 03:49:59.275261 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:49:59.274822 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 03:50:00.013211 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:50:00.013169 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:50:00.271934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.271856 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 03:50:00.272081 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.271924 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 03:50:00.799940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.799902 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq"] Nov 03 03:50:00.804813 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.804791 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.890037 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.889393 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq"] Nov 03 03:50:00.950800 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.950762 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.950800 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.950804 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.951006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.950931 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.951006 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.950989 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.951118 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.951092 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.951182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.951165 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.951225 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.951197 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.951302 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.951286 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:00.951378 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:00.951363 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh64b\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-kube-api-access-sh64b\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.052688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.052591 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.052688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.052657 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.052918 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.052706 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.052918 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.052767 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.052918 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.052793 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.052918 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.052851 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.053130 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.053020 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.053130 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.053026 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh64b\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-kube-api-access-sh64b\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.053252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.053157 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.053252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.053185 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.053906 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.053878 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.054096 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.054072 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.054144 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.054101 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.054211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.054193 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.055631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.055601 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.055736 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.055688 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.115475 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.115409 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.115690 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.115665 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh64b\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-kube-api-access-sh64b\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.122417 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.122387 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:01.559750 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.559721 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq"] Nov 03 03:50:01.560668 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:50:01.560635 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3660a45d_bddf_4804_a5af_d6c9050bfe1f.slice/crio-2b6e0dd6a0575fd15a94ab178fb2cdf499907d3742d1e8db8f15877837d59295 WatchSource:0}: Error finding container 2b6e0dd6a0575fd15a94ab178fb2cdf499907d3742d1e8db8f15877837d59295: Status 404 returned error can't find the container with id 2b6e0dd6a0575fd15a94ab178fb2cdf499907d3742d1e8db8f15877837d59295 Nov 03 03:50:01.562852 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.562819 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:01.562945 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.562884 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:01.562945 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:01.562911 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:02.058272 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.058227 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" event={"ID":"3660a45d-bddf-4804-a5af-d6c9050bfe1f","Type":"ContainerStarted","Data":"f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67"} Nov 03 03:50:02.058725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.058304 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" event={"ID":"3660a45d-bddf-4804-a5af-d6c9050bfe1f","Type":"ContainerStarted","Data":"2b6e0dd6a0575fd15a94ab178fb2cdf499907d3742d1e8db8f15877837d59295"} Nov 03 03:50:02.136471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.125719 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:02.136471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.125856 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 03:50:02.136471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.125903 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 03:50:02.198585 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.198526 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podStartSLOduration=2.198513014 podStartE2EDuration="2.198513014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:02.197205531 +0000 UTC m=+2937.227804070" watchObservedRunningTime="2026-11-03 03:50:02.198513014 +0000 UTC m=+2937.229111552" Nov 03 03:50:02.321472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.317573 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:50:02.321472 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.317646 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:50:02.618504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.616742 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:50:02.618504 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.618253 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:50:02.622760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:02.622692 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:50:03.124544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:03.122902 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 03:50:03.124544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:03.122968 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 03:50:04.124544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:04.123812 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 03:50:04.124544 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:04.123877 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 03:50:05.124827 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:05.124794 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 03:50:05.125232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:05.124849 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 03:50:06.124855 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:06.124813 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 03:50:06.125309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:06.124900 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 03:50:07.124857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:07.124817 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 03:50:07.125293 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:07.124889 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 03:50:08.123841 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:08.123803 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 03:50:08.124047 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:08.123859 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 03:50:09.128314 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:09.128278 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:09.128830 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:09.128744 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:09.129603 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:09.129583 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:50:15.908994 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.908955 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg"] Nov 03 03:50:15.914478 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.914418 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953043 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953006 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkr4\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-kube-api-access-kqkr4\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953054 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953154 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953210 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953316 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96310c43-9954-4cfe-907a-df1272d49305-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953341 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96310c43-9954-4cfe-907a-df1272d49305-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953469 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953415 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953567 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953551 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:15.953629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:15.953607 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.017682 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.017645 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg"] Nov 03 03:50:16.054247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054211 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkr4\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-kube-api-access-kqkr4\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054255 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054565 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054379 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054565 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054437 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054608 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054646 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96310c43-9954-4cfe-907a-df1272d49305-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054802 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054705 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96310c43-9954-4cfe-907a-df1272d49305-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054802 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054713 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054802 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054793 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054994 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054842 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.054994 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054872 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.055106 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.055064 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.055106 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.054843 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.055265 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.055240 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96310c43-9954-4cfe-907a-df1272d49305-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.057273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.057244 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.057553 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.057529 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96310c43-9954-4cfe-907a-df1272d49305-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.116378 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.116336 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.116573 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.116496 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkr4\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-kube-api-access-kqkr4\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.228421 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.228340 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:16.660958 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.660930 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg"] Nov 03 03:50:16.662034 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:50:16.661954 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96310c43_9954_4cfe_907a_df1272d49305.slice/crio-5e8968c5ce34c6fe0f35b207995e92be6851d7649d2c7480fc1f33d44faef7f5 WatchSource:0}: Error finding container 5e8968c5ce34c6fe0f35b207995e92be6851d7649d2c7480fc1f33d44faef7f5: Status 404 returned error can't find the container with id 5e8968c5ce34c6fe0f35b207995e92be6851d7649d2c7480fc1f33d44faef7f5 Nov 03 03:50:16.664711 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.664678 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:16.664814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.664743 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:16.664814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:16.664772 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:17.164541 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:17.164498 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" event={"ID":"96310c43-9954-4cfe-907a-df1272d49305","Type":"ContainerStarted","Data":"449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581"} Nov 03 03:50:17.165604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:17.165562 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" event={"ID":"96310c43-9954-4cfe-907a-df1272d49305","Type":"ContainerStarted","Data":"5e8968c5ce34c6fe0f35b207995e92be6851d7649d2c7480fc1f33d44faef7f5"} Nov 03 03:50:17.228630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:17.228595 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:17.230508 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:17.230485 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 03:50:17.230634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:17.230538 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 03:50:17.293659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:17.293613 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podStartSLOduration=2.293601184 podStartE2EDuration="2.293601184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:17.290855667 +0000 UTC m=+2952.321454207" watchObservedRunningTime="2026-11-03 03:50:17.293601184 +0000 UTC m=+2952.324199722" Nov 03 03:50:18.231781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:18.231744 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 03:50:18.232385 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:18.232352 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 03:50:19.228900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:19.228851 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 03:50:19.229182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:19.229155 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 03:50:20.228982 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:20.228942 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 03:50:20.229404 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:20.229015 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 03:50:21.229182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:21.229146 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 03:50:21.229683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:21.229206 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 03:50:22.229124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:22.229080 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 03:50:22.229339 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:22.229146 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 03:50:23.228781 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:23.228749 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 03:50:23.228975 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:23.228804 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 03:50:24.233043 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:24.233012 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:24.233566 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:24.233402 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:24.234571 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:24.234548 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 03:50:30.014371 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:50:30.014319 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:50:37.944840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.944803 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c"] Nov 03 03:50:37.949117 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.949092 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991239 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991198 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991416 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991247 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991416 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991350 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfd8\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-kube-api-access-fmfd8\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991417 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991480 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991596 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991581 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991627 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991606 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991665 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991624 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:37.991665 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:37.991646 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.009182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.009145 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c"] Nov 03 03:50:38.092655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092613 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.092655 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092657 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.092904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092686 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.092904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092707 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.092904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092724 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.092904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092745 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.092904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092836 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.092904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092867 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.093203 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.092921 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfd8\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-kube-api-access-fmfd8\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.093263 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.093223 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.093321 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.093287 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.093391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.093368 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.093481 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.093387 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.093557 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.093487 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.095205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.095183 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.095479 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.095429 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.154935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.154898 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.155211 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.155182 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfd8\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-kube-api-access-fmfd8\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.263686 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.263588 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:38.697150 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.697111 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c"] Nov 03 03:50:38.698113 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:50:38.698077 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b5de7a_3fb2_4bb3_8e5a_eacbb7f6897b.slice/crio-4c078706eb7ae3806d7a39baf4f71d9b824f8edcfb9a52d353ca9dcd69a97234 WatchSource:0}: Error finding container 4c078706eb7ae3806d7a39baf4f71d9b824f8edcfb9a52d353ca9dcd69a97234: Status 404 returned error can't find the container with id 4c078706eb7ae3806d7a39baf4f71d9b824f8edcfb9a52d353ca9dcd69a97234 Nov 03 03:50:38.700698 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.700659 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:38.700818 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.700745 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:38.700818 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:38.700793 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:50:39.634844 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:39.634809 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" event={"ID":"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b","Type":"ContainerStarted","Data":"43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee"} Nov 03 03:50:39.635280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:39.634860 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" event={"ID":"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b","Type":"ContainerStarted","Data":"4c078706eb7ae3806d7a39baf4f71d9b824f8edcfb9a52d353ca9dcd69a97234"} Nov 03 03:50:40.264696 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:40.264661 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:40.266463 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:40.266379 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:50:40.266609 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:40.266465 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 03:50:41.264310 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:41.264276 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:50:41.264767 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:41.264328 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 03:50:42.264547 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:42.264508 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:50:42.264941 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:42.264579 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 03:50:43.264268 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:43.264231 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:50:43.264422 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:43.264295 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 03:50:44.264665 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:44.264622 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:50:44.265090 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:44.264683 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 03:50:44.748289 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:44.748217 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podStartSLOduration=7.748195894 podStartE2EDuration="7.748195894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:39.826789675 +0000 UTC m=+2974.857388214" watchObservedRunningTime="2026-11-03 03:50:44.748195894 +0000 UTC m=+2979.778794435" Nov 03 03:50:44.749066 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:44.749044 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w"] Nov 03 03:50:44.749279 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:44.749255 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" podUID="afc59226-544a-4420-be01-394f2654df9b" containerName="maas-api" containerID="cri-o://df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5" gracePeriod=30 Nov 03 03:50:45.157743 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.157708 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:50:45.264125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.264075 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 03:50:45.264316 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.264144 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 03:50:45.292571 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.292531 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc59226-544a-4420-be01-394f2654df9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc59226-544a-4420-be01-394f2654df9b-openshift-service-ca\") pod \"afc59226-544a-4420-be01-394f2654df9b\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " Nov 03 03:50:45.293085 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.292596 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls\") pod \"afc59226-544a-4420-be01-394f2654df9b\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " Nov 03 03:50:45.293085 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.292765 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc59226-544a-4420-be01-394f2654df9b-kube-api-access-5m8q2\" (UniqueName: \"kubernetes.io/projected/afc59226-544a-4420-be01-394f2654df9b-kube-api-access-5m8q2\") pod \"afc59226-544a-4420-be01-394f2654df9b\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " Nov 03 03:50:45.293085 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.292820 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-metrics-certs\") pod \"afc59226-544a-4420-be01-394f2654df9b\" (UID: \"afc59226-544a-4420-be01-394f2654df9b\") " Nov 03 03:50:45.293639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.293583 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc59226-544a-4420-be01-394f2654df9b-openshift-service-ca" pod "afc59226-544a-4420-be01-394f2654df9b" (UID: "afc59226-544a-4420-be01-394f2654df9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:45.295898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.295861 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-metrics-certs" pod "afc59226-544a-4420-be01-394f2654df9b" (UID: "afc59226-544a-4420-be01-394f2654df9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:45.296017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.295998 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc59226-544a-4420-be01-394f2654df9b-kube-api-access-5m8q2" pod "afc59226-544a-4420-be01-394f2654df9b" (UID: "afc59226-544a-4420-be01-394f2654df9b"). InnerVolumeSpecName "kube-api-access-5m8q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:45.298806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.298770 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls" pod "afc59226-544a-4420-be01-394f2654df9b" (UID: "afc59226-544a-4420-be01-394f2654df9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:45.394273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.394163 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:45.394273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.394196 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m8q2\" (UniqueName: \"kubernetes.io/projected/afc59226-544a-4420-be01-394f2654df9b-kube-api-access-5m8q2\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:45.394273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.394208 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc59226-544a-4420-be01-394f2654df9b-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:45.394273 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.394220 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc59226-544a-4420-be01-394f2654df9b-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:45.674406 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.674304 5117 generic.go:356] "Generic (PLEG): container finished" podID="afc59226-544a-4420-be01-394f2654df9b" containerID="df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5" exitCode=0 Nov 03 03:50:45.674406 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.674350 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" Nov 03 03:50:45.674406 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.674390 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" event={"ID":"afc59226-544a-4420-be01-394f2654df9b","Type":"ContainerDied","Data":"df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5"} Nov 03 03:50:45.674725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.674481 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w" event={"ID":"afc59226-544a-4420-be01-394f2654df9b","Type":"ContainerDied","Data":"8751539eccf52c9426ba27097dafae70ff33da78fdbd0480722a7e0d9eeea94b"} Nov 03 03:50:45.674725 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.674515 5117 scope.go:122] "RemoveContainer" containerID="df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5" Nov 03 03:50:45.686804 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.686782 5117 scope.go:122] "RemoveContainer" containerID="df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5" Nov 03 03:50:45.687087 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:50:45.687067 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5\": container with ID starting with df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5 not found: ID does not exist" containerID="df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5" Nov 03 03:50:45.687394 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.687096 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5"} err="failed to get container status \"df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5\": rpc error: code = NotFound desc = could not find container \"df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5\": container with ID starting with df01681affece02217edde9f0ecec8af0d6db3c147c56f2b01dd715f8db27cd5 not found: ID does not exist" Nov 03 03:50:45.845360 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.845322 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w"] Nov 03 03:50:45.849741 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.849716 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-29dd3f-7957b944c9-4dq9w"] Nov 03 03:50:45.917160 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.917120 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8"] Nov 03 03:50:45.917328 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:45.917303 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" podUID="9a4d205e-f332-4353-ba6c-e79e24aede56" containerName="maas-api" containerID="cri-o://cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d" gracePeriod=30 Nov 03 03:50:46.256071 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.255767 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:50:46.333110 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.333071 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4d205e-f332-4353-ba6c-e79e24aede56-kube-api-access-97wvg\" (UniqueName: \"kubernetes.io/projected/9a4d205e-f332-4353-ba6c-e79e24aede56-kube-api-access-97wvg\") pod \"9a4d205e-f332-4353-ba6c-e79e24aede56\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " Nov 03 03:50:46.333629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.333188 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls\") pod \"9a4d205e-f332-4353-ba6c-e79e24aede56\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " Nov 03 03:50:46.333629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.333285 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-metrics-certs\") pod \"9a4d205e-f332-4353-ba6c-e79e24aede56\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " Nov 03 03:50:46.333629 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.333396 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4d205e-f332-4353-ba6c-e79e24aede56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d205e-f332-4353-ba6c-e79e24aede56-openshift-service-ca\") pod \"9a4d205e-f332-4353-ba6c-e79e24aede56\" (UID: \"9a4d205e-f332-4353-ba6c-e79e24aede56\") " Nov 03 03:50:46.333897 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.333870 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4d205e-f332-4353-ba6c-e79e24aede56-openshift-service-ca" pod "9a4d205e-f332-4353-ba6c-e79e24aede56" (UID: "9a4d205e-f332-4353-ba6c-e79e24aede56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:50:46.335807 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.335784 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4d205e-f332-4353-ba6c-e79e24aede56-kube-api-access-97wvg" pod "9a4d205e-f332-4353-ba6c-e79e24aede56" (UID: "9a4d205e-f332-4353-ba6c-e79e24aede56"). InnerVolumeSpecName "kube-api-access-97wvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:50:46.336018 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.335994 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-metrics-certs" pod "9a4d205e-f332-4353-ba6c-e79e24aede56" (UID: "9a4d205e-f332-4353-ba6c-e79e24aede56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:46.336854 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.336816 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls" pod "9a4d205e-f332-4353-ba6c-e79e24aede56" (UID: "9a4d205e-f332-4353-ba6c-e79e24aede56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:50:46.367464 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.365054 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:46.367464 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.366019 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:46.370480 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.369260 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:50:46.435529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.435392 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:46.435529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.435428 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d205e-f332-4353-ba6c-e79e24aede56-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:46.435529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.435485 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wvg\" (UniqueName: \"kubernetes.io/projected/9a4d205e-f332-4353-ba6c-e79e24aede56-kube-api-access-97wvg\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:46.435529 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.435502 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4d205e-f332-4353-ba6c-e79e24aede56-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:50:46.682712 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.682671 5117 generic.go:356] "Generic (PLEG): container finished" podID="9a4d205e-f332-4353-ba6c-e79e24aede56" containerID="cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d" exitCode=0 Nov 03 03:50:46.682907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.682726 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" Nov 03 03:50:46.682907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.682758 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" event={"ID":"9a4d205e-f332-4353-ba6c-e79e24aede56","Type":"ContainerDied","Data":"cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d"} Nov 03 03:50:46.682907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.682815 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8" event={"ID":"9a4d205e-f332-4353-ba6c-e79e24aede56","Type":"ContainerDied","Data":"29d1ea99cae1fa1386ec8c982dd6825b0e61ba002ea00233ace9866c89e9c3cb"} Nov 03 03:50:46.682907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.682837 5117 scope.go:122] "RemoveContainer" containerID="cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d" Nov 03 03:50:46.694481 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.694458 5117 scope.go:122] "RemoveContainer" containerID="cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d" Nov 03 03:50:46.694776 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:50:46.694755 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d\": container with ID starting with cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d not found: ID does not exist" containerID="cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d" Nov 03 03:50:46.694846 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.694791 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d"} err="failed to get container status \"cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d\": rpc error: code = NotFound desc = could not find container \"cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d\": container with ID starting with cbfd4ab10b55dc74e8cb80e16f18d273bb8fd86a88fee7a419b7cbd3aaca561d not found: ID does not exist" Nov 03 03:50:46.850890 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.850837 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8"] Nov 03 03:50:46.853389 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:46.853361 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5ec4ce-76568db5c8-cxhz8"] Nov 03 03:50:47.090722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:47.090681 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4d205e-f332-4353-ba6c-e79e24aede56" path="/var/lib/kubelet/pods/9a4d205e-f332-4353-ba6c-e79e24aede56/volumes" Nov 03 03:50:47.091279 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:50:47.091244 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc59226-544a-4420-be01-394f2654df9b" path="/var/lib/kubelet/pods/afc59226-544a-4420-be01-394f2654df9b/volumes" Nov 03 03:51:00.011615 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:51:00.011571 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:51:07.566210 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.566175 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:51:07.574183 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.574162 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:51:07.968509 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.968404 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b"] Nov 03 03:51:07.968954 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.968940 5117 cpu_manager.go:414] "containerMap: removing container" podUID="9a4d205e-f332-4353-ba6c-e79e24aede56" containerName="maas-api" Nov 03 03:51:07.968997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.968956 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4d205e-f332-4353-ba6c-e79e24aede56" containerName="maas-api" Nov 03 03:51:07.968997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.968972 5117 cpu_manager.go:414] "containerMap: removing container" podUID="afc59226-544a-4420-be01-394f2654df9b" containerName="maas-api" Nov 03 03:51:07.968997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.968976 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc59226-544a-4420-be01-394f2654df9b" containerName="maas-api" Nov 03 03:51:07.969099 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.969080 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc59226-544a-4420-be01-394f2654df9b" containerName="maas-api" Nov 03 03:51:07.969099 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.969097 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4d205e-f332-4353-ba6c-e79e24aede56" containerName="maas-api" Nov 03 03:51:07.973757 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:07.973738 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.027679 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.027637 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5a6246-openshift-default-dockercfg-nmbkv\"" Nov 03 03:51:08.081744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.081706 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b"] Nov 03 03:51:08.108717 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.108675 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d0d93-538e-498a-a3dc-b819b663eb30-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.108917 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.108826 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d0d93-538e-498a-a3dc-b819b663eb30-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.108917 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.108861 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.109041 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.108981 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.109041 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.109021 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.109128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.109051 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.109163 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.109141 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7f2r\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-kube-api-access-w7f2r\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.109197 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.109179 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.109234 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.109198 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.209921 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.209888 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.209974 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.209995 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210009 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210069 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7f2r\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-kube-api-access-w7f2r\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210128 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210110 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210503 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210139 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210503 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210174 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d0d93-538e-498a-a3dc-b819b663eb30-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210503 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210410 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210501 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210509 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d0d93-538e-498a-a3dc-b819b663eb30-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210514 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210919 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.210951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.210923 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d0d93-538e-498a-a3dc-b819b663eb30-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.212607 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.212584 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.213232 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.213209 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d0d93-538e-498a-a3dc-b819b663eb30-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.273057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.272983 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.273057 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.272985 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7f2r\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-kube-api-access-w7f2r\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.287934 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.287905 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:08.735572 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.735545 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b"] Nov 03 03:51:08.737225 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:51:08.737195 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604d0d93_538e_498a_a3dc_b819b663eb30.slice/crio-4442063e88744706761c33e1c4038aa7534094875433f6c9c3a126a0fb753831 WatchSource:0}: Error finding container 4442063e88744706761c33e1c4038aa7534094875433f6c9c3a126a0fb753831: Status 404 returned error can't find the container with id 4442063e88744706761c33e1c4038aa7534094875433f6c9c3a126a0fb753831 Nov 03 03:51:08.739756 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.739723 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:51:08.739834 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.739786 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:51:08.739834 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.739815 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 03:51:08.913574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.912778 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" event={"ID":"604d0d93-538e-498a-a3dc-b819b663eb30","Type":"ContainerStarted","Data":"e4325def31f36a42e3d1f962fec234896870802181a7f3cf2b592d4fa9d8c1a2"} Nov 03 03:51:08.913574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:08.912856 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" event={"ID":"604d0d93-538e-498a-a3dc-b819b663eb30","Type":"ContainerStarted","Data":"4442063e88744706761c33e1c4038aa7534094875433f6c9c3a126a0fb753831"} Nov 03 03:51:09.040316 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:09.040187 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podStartSLOduration=2.040170206 podStartE2EDuration="2.040170206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:09.0384399 +0000 UTC m=+3004.069038439" watchObservedRunningTime="2026-11-03 03:51:09.040170206 +0000 UTC m=+3004.070768746" Nov 03 03:51:09.294540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:09.288879 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:09.294540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:09.290919 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 03:51:09.294540 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:09.290969 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 03:51:10.288338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:10.288299 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 03:51:10.289506 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:10.288362 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 03:51:11.295280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:11.292925 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 03:51:11.295280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:11.293018 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 03:51:11.984113 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:11.984031 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 03:51:11.984113 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:11.984072 5117 generic.go:356] "Generic (PLEG): container finished" podID="2469c20c-30a7-41bb-b726-d62d9324f693" containerID="e2b3c6138b1a104cfce552df554a805910a738cd55f99157b9bda0cd058b432b" exitCode=2 Nov 03 03:51:11.984301 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:11.984137 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" event={"ID":"2469c20c-30a7-41bb-b726-d62d9324f693","Type":"ContainerDied","Data":"e2b3c6138b1a104cfce552df554a805910a738cd55f99157b9bda0cd058b432b"} Nov 03 03:51:11.984559 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:11.984544 5117 scope.go:122] "RemoveContainer" containerID="e2b3c6138b1a104cfce552df554a805910a738cd55f99157b9bda0cd058b432b" Nov 03 03:51:12.289115 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:12.289037 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 03:51:12.289115 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:12.289091 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 03:51:12.993826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:12.993796 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 03:51:12.994222 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:12.993869 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6t2pr" event={"ID":"2469c20c-30a7-41bb-b726-d62d9324f693","Type":"ContainerStarted","Data":"69b269082de8710aba592df06038d5098a2fd9e971ebc01284a3ab0505d3fec9"} Nov 03 03:51:13.289093 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:13.289002 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 03:51:13.289420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:13.289388 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 03:51:14.288567 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:14.288525 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 03:51:14.289014 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:14.288597 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 03:51:15.288887 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:15.288842 5117 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 03:51:15.289318 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:15.288916 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 03:51:16.293322 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:16.293281 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:16.294047 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:16.294022 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:16.294834 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:16.294816 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 03:51:21.831468 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:21.831313 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 03:51:21.998471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:21.982366 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" podUID="6735ef43-0f84-4832-b8be-6947a0401721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 03 03:51:22.215648 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:22.215536 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" podUID="38375563-5dc0-4101-93af-ad9978d8997b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.233518 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:22.233476 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 03:51:22.447471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:22.446542 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.491637 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:22.491538 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.990331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:22.990285 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" podUID="6735ef43-0f84-4832-b8be-6947a0401721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:22.991358 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:22.991334 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:51:23.071179 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:23.071149 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 03:51:23.260559 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:23.248500 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 03:51:23.301766 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:23.301287 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:51:23.336312 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:23.336199 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 03:51:30.011978 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:51:30.011938 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:51:43.827636 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.827603 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j"] Nov 03 03:51:43.832851 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.832829 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:43.887493 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.887458 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-10120d-serving-cert\"" Nov 03 03:51:43.937267 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.937228 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:43.937478 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.937383 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:43.937615 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.937597 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ts7r\" (UniqueName: \"kubernetes.io/projected/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-kube-api-access-8ts7r\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:43.937685 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.937662 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:43.941157 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:43.941127 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j"] Nov 03 03:51:44.038701 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.038670 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ts7r\" (UniqueName: \"kubernetes.io/projected/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-kube-api-access-8ts7r\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.038889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.038739 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.038889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.038804 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.038889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.038884 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.039055 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:51:44.038922 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-serving-cert: secret "maas-api-e2e-shared-a-w1-10120d-serving-cert" not found Nov 03 03:51:44.039055 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:51:44.038993 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls podName:51f3a7f5-ca55-471e-b18c-1bd44a8c1198 nodeName:}" failed. No retries permitted until 2026-11-03 03:51:44.538972912 +0000 UTC m=+3039.569571430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls") pod "maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" (UID: "51f3a7f5-ca55-471e-b18c-1bd44a8c1198") : secret "maas-api-e2e-shared-a-w1-10120d-serving-cert" not found Nov 03 03:51:44.039537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.039514 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.041773 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.041746 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.103405 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.103321 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ts7r\" (UniqueName: \"kubernetes.io/projected/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-kube-api-access-8ts7r\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.545639 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.545537 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.548353 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.548321 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:44.747052 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:44.747012 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:45.127112 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:45.127086 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j"] Nov 03 03:51:45.128642 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:51:45.128603 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f3a7f5_ca55_471e_b18c_1bd44a8c1198.slice/crio-6b9cb974cadcf3938bd8586575be92849dfb1c94f52ca3b47888d0450b0171d5 WatchSource:0}: Error finding container 6b9cb974cadcf3938bd8586575be92849dfb1c94f52ca3b47888d0450b0171d5: Status 404 returned error can't find the container with id 6b9cb974cadcf3938bd8586575be92849dfb1c94f52ca3b47888d0450b0171d5 Nov 03 03:51:45.130319 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:45.130298 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:51:45.399957 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:45.399857 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" event={"ID":"51f3a7f5-ca55-471e-b18c-1bd44a8c1198","Type":"ContainerStarted","Data":"6b9cb974cadcf3938bd8586575be92849dfb1c94f52ca3b47888d0450b0171d5"} Nov 03 03:51:46.408125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:46.407836 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" event={"ID":"51f3a7f5-ca55-471e-b18c-1bd44a8c1198","Type":"ContainerStarted","Data":"1f55e600edc883dae90c293d4703398030e43733d0e7ed75734cf9500b5fb2db"} Nov 03 03:51:46.408948 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:46.408926 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:46.528710 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:46.528651 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" podStartSLOduration=2.638956977 podStartE2EDuration="3.528637787s" totalImagesPullingTime="889.68081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:46.528133155 +0000 UTC m=+3041.558731690" watchObservedRunningTime="2026-11-03 03:51:46.528637787 +0000 UTC m=+3041.559236341" Nov 03 03:51:52.419941 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:52.419908 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 03:51:53.971046 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:53.971014 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9"] Nov 03 03:51:53.975237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:53.975221 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.031026 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.030992 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-66d902-serving-cert\"" Nov 03 03:51:54.057903 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.057863 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c357377-c079-42a9-88d5-ecdf1e32504b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.058074 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.057922 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.058074 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.058041 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.058074 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.058063 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72lbs\" (UniqueName: \"kubernetes.io/projected/1c357377-c079-42a9-88d5-ecdf1e32504b-kube-api-access-72lbs\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.085141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.085110 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9"] Nov 03 03:51:54.159007 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.158972 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c357377-c079-42a9-88d5-ecdf1e32504b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.159187 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.159038 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.159187 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.159126 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.159187 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.159155 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lbs\" (UniqueName: \"kubernetes.io/projected/1c357377-c079-42a9-88d5-ecdf1e32504b-kube-api-access-72lbs\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.159359 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:51:54.159199 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-serving-cert: secret "maas-api-e2e-shared-a-w2-66d902-serving-cert" not found Nov 03 03:51:54.159359 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:51:54.159270 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls podName:1c357377-c079-42a9-88d5-ecdf1e32504b nodeName:}" failed. No retries permitted until 2026-11-03 03:51:54.659248934 +0000 UTC m=+3049.689847467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" (UID: "1c357377-c079-42a9-88d5-ecdf1e32504b") : secret "maas-api-e2e-shared-a-w2-66d902-serving-cert" not found Nov 03 03:51:54.159722 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.159695 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c357377-c079-42a9-88d5-ecdf1e32504b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.161673 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.161652 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.222063 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.221995 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lbs\" (UniqueName: \"kubernetes.io/projected/1c357377-c079-42a9-88d5-ecdf1e32504b-kube-api-access-72lbs\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.664724 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.664690 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.667210 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.667174 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:54.888721 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:54.888683 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:55.271862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:55.271830 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9"] Nov 03 03:51:55.277752 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:51:55.277720 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c357377_c079_42a9_88d5_ecdf1e32504b.slice/crio-2aa0420481960d6a8bb5c087cb0ec2a66ad5b71a99883022796b8341eb833b55 WatchSource:0}: Error finding container 2aa0420481960d6a8bb5c087cb0ec2a66ad5b71a99883022796b8341eb833b55: Status 404 returned error can't find the container with id 2aa0420481960d6a8bb5c087cb0ec2a66ad5b71a99883022796b8341eb833b55 Nov 03 03:51:55.463011 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:55.462970 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" event={"ID":"1c357377-c079-42a9-88d5-ecdf1e32504b","Type":"ContainerStarted","Data":"2aa0420481960d6a8bb5c087cb0ec2a66ad5b71a99883022796b8341eb833b55"} Nov 03 03:51:56.471019 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:56.470929 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" event={"ID":"1c357377-c079-42a9-88d5-ecdf1e32504b","Type":"ContainerStarted","Data":"083d4db5eb49845aa21211bea69f19f349f7dff720ee51465cfac4371a4f8b2f"} Nov 03 03:51:56.471393 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:56.471134 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:51:56.592684 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:51:56.592621 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" podStartSLOduration=2.715099438 podStartE2EDuration="3.592605722s" totalImagesPullingTime="877.506284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:56.590297973 +0000 UTC m=+3051.620896512" watchObservedRunningTime="2026-11-03 03:51:56.592605722 +0000 UTC m=+3051.623204259" Nov 03 03:52:00.012266 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:52:00.012224 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:52:02.484494 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:02.484461 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 03:52:30.018065 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:52:30.018010 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:52:33.551707 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.551672 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n"] Nov 03 03:52:33.556996 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.556965 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.616039 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.616001 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-metrics-certs\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.616254 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.616048 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.616254 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.616162 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwcmm\" (UniqueName: \"kubernetes.io/projected/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-kube-api-access-fwcmm\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.616254 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.616214 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-openshift-service-ca\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.664824 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.664785 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-158d23-serving-cert\"" Nov 03 03:52:33.669554 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.669519 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n"] Nov 03 03:52:33.717470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.717409 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-metrics-certs\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.717470 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.717473 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.717666 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.717504 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwcmm\" (UniqueName: \"kubernetes.io/projected/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-kube-api-access-fwcmm\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.717666 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.717549 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-openshift-service-ca\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.717666 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:52:33.717606 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-serving-cert: secret "maas-api-e2e-explicit-158d23-serving-cert" not found Nov 03 03:52:33.717782 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:52:33.717676 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls podName:70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e nodeName:}" failed. No retries permitted until 2026-11-03 03:52:34.217656414 +0000 UTC m=+3089.248254948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls") pod "maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" (UID: "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e") : secret "maas-api-e2e-explicit-158d23-serving-cert" not found Nov 03 03:52:33.718144 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.718124 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-openshift-service-ca\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.720247 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.720220 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-metrics-certs\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:33.779692 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:33.779661 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwcmm\" (UniqueName: \"kubernetes.io/projected/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-kube-api-access-fwcmm\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:34.222471 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:34.222416 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:34.225024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:34.225005 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls\") pod \"maas-api-e2e-explicit-158d23-865fb6567c-9zs7n\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:34.475691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:34.475612 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:34.857237 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:34.857207 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n"] Nov 03 03:52:34.857748 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:52:34.857724 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d4b3dd_8f85_4dd9_a4c0_46f0b0a3937e.slice/crio-c3de9efbc59e2721925f38b641551b9938a6d513663cbf5ad8cb26c0d92d517b WatchSource:0}: Error finding container c3de9efbc59e2721925f38b641551b9938a6d513663cbf5ad8cb26c0d92d517b: Status 404 returned error can't find the container with id c3de9efbc59e2721925f38b641551b9938a6d513663cbf5ad8cb26c0d92d517b Nov 03 03:52:35.713095 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:35.713042 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" event={"ID":"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e","Type":"ContainerStarted","Data":"c3de9efbc59e2721925f38b641551b9938a6d513663cbf5ad8cb26c0d92d517b"} Nov 03 03:52:36.721157 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:36.721111 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" event={"ID":"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e","Type":"ContainerStarted","Data":"1c8b24cfd05a5da1c28dfe92f6b828274a0c0b99152b429d1010159a7f451ce6"} Nov 03 03:52:36.721674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:36.721207 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:52:36.851029 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:36.850973 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" podStartSLOduration=3.026078162 podStartE2EDuration="3.850961053s" totalImagesPullingTime="824.882891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:36.850085968 +0000 UTC m=+3091.880684508" watchObservedRunningTime="2026-11-03 03:52:36.850961053 +0000 UTC m=+3091.881559591" Nov 03 03:52:42.734391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:52:42.734314 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:53:00.013037 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:53:00.012989 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:53:30.011598 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:53:30.011559 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:53:36.001288 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.001250 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf"] Nov 03 03:53:36.005625 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.005606 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.048199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.048174 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h482\" (UniqueName: \"kubernetes.io/projected/b4e5fc07-68de-42ef-8737-354f5fee1710-kube-api-access-2h482\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.048314 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.048230 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.048314 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.048309 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.048393 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.048353 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e5fc07-68de-42ef-8737-354f5fee1710-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.094323 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.094275 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-7ed2d6-serving-cert\"" Nov 03 03:53:36.097569 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.097540 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf"] Nov 03 03:53:36.149562 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.149530 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h482\" (UniqueName: \"kubernetes.io/projected/b4e5fc07-68de-42ef-8737-354f5fee1710-kube-api-access-2h482\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.149828 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.149800 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.149954 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.149843 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.149954 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.149902 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e5fc07-68de-42ef-8737-354f5fee1710-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.154584 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:53:36.150297 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-serving-cert: secret "maas-api-e2e-shared-b-w4-7ed2d6-serving-cert" not found Nov 03 03:53:36.154584 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:53:36.150368 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls podName:b4e5fc07-68de-42ef-8737-354f5fee1710 nodeName:}" failed. No retries permitted until 2026-11-03 03:53:36.650345526 +0000 UTC m=+3151.680944047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls") pod "maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" (UID: "b4e5fc07-68de-42ef-8737-354f5fee1710") : secret "maas-api-e2e-shared-b-w4-7ed2d6-serving-cert" not found Nov 03 03:53:36.154584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.150827 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e5fc07-68de-42ef-8737-354f5fee1710-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.154584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.152634 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.214884 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.214847 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h482\" (UniqueName: \"kubernetes.io/projected/b4e5fc07-68de-42ef-8737-354f5fee1710-kube-api-access-2h482\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.657425 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.657393 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.659974 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.659958 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:36.919255 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:36.919161 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:37.296198 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:37.296169 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf"] Nov 03 03:53:37.296965 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:53:37.296922 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e5fc07_68de_42ef_8737_354f5fee1710.slice/crio-dbc3a3d098be26354ca718546d1da016522360c4882b379a20d48e71d294fbc1 WatchSource:0}: Error finding container dbc3a3d098be26354ca718546d1da016522360c4882b379a20d48e71d294fbc1: Status 404 returned error can't find the container with id dbc3a3d098be26354ca718546d1da016522360c4882b379a20d48e71d294fbc1 Nov 03 03:53:38.110826 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:38.110788 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" event={"ID":"b4e5fc07-68de-42ef-8737-354f5fee1710","Type":"ContainerStarted","Data":"dbc3a3d098be26354ca718546d1da016522360c4882b379a20d48e71d294fbc1"} Nov 03 03:53:39.119191 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:39.119153 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" event={"ID":"b4e5fc07-68de-42ef-8737-354f5fee1710","Type":"ContainerStarted","Data":"314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab"} Nov 03 03:53:39.119728 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:39.119226 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:39.240294 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:39.240228 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" podStartSLOduration=3.24937767 podStartE2EDuration="4.240213673s" totalImagesPullingTime="990.836003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:39.238719486 +0000 UTC m=+3154.269318025" watchObservedRunningTime="2026-11-03 03:53:39.240213673 +0000 UTC m=+3154.270812210" Nov 03 03:53:42.061844 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.061804 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8"] Nov 03 03:53:42.066141 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.066114 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.122617 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.122571 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tls-certs\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.122815 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.122712 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-model-cache\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.122815 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.122742 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz8k\" (UniqueName: \"kubernetes.io/projected/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-kube-api-access-4vz8k\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.122954 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.122841 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-home\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.122954 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.122887 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tmp-dir\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.122954 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.122918 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-dshm\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.142995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.142967 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-158d23\"/\"openshift-service-ca.crt\"" Nov 03 03:53:42.143260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.143238 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-158d23\"/\"default-dockercfg-qldhb\"" Nov 03 03:53:42.143260 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.143251 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-158d23\"/\"kube-root-ca.crt\"" Nov 03 03:53:42.171280 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.171254 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-158d23\"/\"explicit-model-158d23-kserve-self-signed-certs\"" Nov 03 03:53:42.174342 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.174318 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8"] Nov 03 03:53:42.223975 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.223925 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-model-cache\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.224395 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.224279 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz8k\" (UniqueName: \"kubernetes.io/projected/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-kube-api-access-4vz8k\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.224569 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.224546 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-home\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.224642 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.224604 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tmp-dir\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.224703 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.224639 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-dshm\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.224768 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.224730 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tls-certs\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.227965 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.227924 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tmp-dir\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.229952 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.229898 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-dshm\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.233220 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.233160 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tls-certs\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.238274 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.238246 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-model-cache\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.238648 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.238628 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-home\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.290181 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.290148 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz8k\" (UniqueName: \"kubernetes.io/projected/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-kube-api-access-4vz8k\") pod \"explicit-model-158d23-kserve-657cb7cf89-pr4q8\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.379509 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.379373 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:42.763887 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:42.763853 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8"] Nov 03 03:53:42.764551 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:53:42.764523 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91c5a36e_446f_4b4d_9ca5_0bae41b084a7.slice/crio-7214e9c5de4c34b9851364a1b1a829f566b3c787090e10172649db98d2c6e013 WatchSource:0}: Error finding container 7214e9c5de4c34b9851364a1b1a829f566b3c787090e10172649db98d2c6e013: Status 404 returned error can't find the container with id 7214e9c5de4c34b9851364a1b1a829f566b3c787090e10172649db98d2c6e013 Nov 03 03:53:43.146501 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:43.146437 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" event={"ID":"91c5a36e-446f-4b4d-9ca5-0bae41b084a7","Type":"ContainerStarted","Data":"bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555"} Nov 03 03:53:43.146904 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:43.146512 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" event={"ID":"91c5a36e-446f-4b4d-9ca5-0bae41b084a7","Type":"ContainerStarted","Data":"7214e9c5de4c34b9851364a1b1a829f566b3c787090e10172649db98d2c6e013"} Nov 03 03:53:43.267456 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:43.267394 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" podStartSLOduration=1.26738043 podStartE2EDuration="1.26738043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:43.26613455 +0000 UTC m=+3158.296733089" watchObservedRunningTime="2026-11-03 03:53:43.26738043 +0000 UTC m=+3158.297979013" Nov 03 03:53:43.380805 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:43.380722 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:45.130736 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:45.130704 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 03:53:51.169579 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.169541 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct"] Nov 03 03:53:51.173723 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.173701 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.240517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.236385 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47833055-fc67-4320-b300-05193b9dd528-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.240517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.236539 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.240517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.236598 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.240517 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.236706 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4czg\" (UniqueName: \"kubernetes.io/projected/47833055-fc67-4320-b300-05193b9dd528-kube-api-access-m4czg\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.279392 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.279342 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-9daccc-serving-cert\"" Nov 03 03:53:51.282925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.282896 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct"] Nov 03 03:53:51.337710 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.337668 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47833055-fc67-4320-b300-05193b9dd528-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.337900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.337725 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.337900 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:53:51.337824 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-serving-cert: secret "maas-api-e2e-shared-b-w1-9daccc-serving-cert" not found Nov 03 03:53:51.337900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.337822 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.338062 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:53:51.337908 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls podName:47833055-fc67-4320-b300-05193b9dd528 nodeName:}" failed. No retries permitted until 2026-11-03 03:53:51.837885877 +0000 UTC m=+3166.868484407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls") pod "maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" (UID: "47833055-fc67-4320-b300-05193b9dd528") : secret "maas-api-e2e-shared-b-w1-9daccc-serving-cert" not found Nov 03 03:53:51.338062 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.337940 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4czg\" (UniqueName: \"kubernetes.io/projected/47833055-fc67-4320-b300-05193b9dd528-kube-api-access-m4czg\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.338438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.338419 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47833055-fc67-4320-b300-05193b9dd528-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.340604 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.340584 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.400122 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.400087 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4czg\" (UniqueName: \"kubernetes.io/projected/47833055-fc67-4320-b300-05193b9dd528-kube-api-access-m4czg\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.843820 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.843782 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:51.846352 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:51.846325 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:52.088546 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:52.088513 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:53:52.380159 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:52.380081 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:52.392911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:52.392884 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:52.467268 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:53:52.467237 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47833055_fc67_4320_b300_05193b9dd528.slice/crio-594e4532ed9132418984ed5402cdc2b19fbe4797ae333261b5177e865dc50edc WatchSource:0}: Error finding container 594e4532ed9132418984ed5402cdc2b19fbe4797ae333261b5177e865dc50edc: Status 404 returned error can't find the container with id 594e4532ed9132418984ed5402cdc2b19fbe4797ae333261b5177e865dc50edc Nov 03 03:53:52.500366 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:52.500337 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct"] Nov 03 03:53:53.221069 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:53.221025 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" event={"ID":"47833055-fc67-4320-b300-05193b9dd528","Type":"ContainerStarted","Data":"594e4532ed9132418984ed5402cdc2b19fbe4797ae333261b5177e865dc50edc"} Nov 03 03:53:53.238092 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:53.238061 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:53:54.232790 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:54.232747 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" event={"ID":"47833055-fc67-4320-b300-05193b9dd528","Type":"ContainerStarted","Data":"37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4"} Nov 03 03:53:54.354076 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:54.354024 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" podStartSLOduration=2.506088663 podStartE2EDuration="3.354007303s" totalImagesPullingTime="847.91864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:54.352214443 +0000 UTC m=+3169.382812982" watchObservedRunningTime="2026-11-03 03:53:54.354007303 +0000 UTC m=+3169.384605844" Nov 03 03:53:55.239357 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:53:55.239327 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:54:00.012156 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:54:00.012109 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:54:01.250950 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:01.250919 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 03:54:10.750149 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.750071 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr"] Nov 03 03:54:10.758857 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.758823 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.815576 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.815545 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr"] Nov 03 03:54:10.862636 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.862602 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-3d8dc5-serving-cert\"" Nov 03 03:54:10.875286 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.875259 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.875428 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.875295 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.875498 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.875465 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzlvh\" (UniqueName: \"kubernetes.io/projected/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-kube-api-access-hzlvh\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.875575 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.875559 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.976568 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.976532 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.976750 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.976647 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.976750 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.976671 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.976750 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.976749 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzlvh\" (UniqueName: \"kubernetes.io/projected/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-kube-api-access-hzlvh\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.977277 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.977252 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.979309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.979287 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:10.979309 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:10.979299 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:11.039579 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:11.039509 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzlvh\" (UniqueName: \"kubernetes.io/projected/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-kube-api-access-hzlvh\") pod \"maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:11.073758 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:11.073614 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:11.454602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:11.454572 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr"] Nov 03 03:54:11.454922 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:54:11.454896 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bcaf2a_8d5b_4ff3_bff7_fed8c902c5a3.slice/crio-4bbbe212edd81fe5be8ffcbe3b5318971ef8c49b2a5efe8d4d9db1049ffad258 WatchSource:0}: Error finding container 4bbbe212edd81fe5be8ffcbe3b5318971ef8c49b2a5efe8d4d9db1049ffad258: Status 404 returned error can't find the container with id 4bbbe212edd81fe5be8ffcbe3b5318971ef8c49b2a5efe8d4d9db1049ffad258 Nov 03 03:54:12.353835 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:12.353806 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" event={"ID":"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3","Type":"ContainerStarted","Data":"4bbbe212edd81fe5be8ffcbe3b5318971ef8c49b2a5efe8d4d9db1049ffad258"} Nov 03 03:54:13.361742 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:13.361706 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" event={"ID":"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3","Type":"ContainerStarted","Data":"dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d"} Nov 03 03:54:13.362227 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:13.361784 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:13.483671 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:13.483619 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" podStartSLOduration=2.604652959 podStartE2EDuration="3.483603345s" totalImagesPullingTime="878.950386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:54:13.481120521 +0000 UTC m=+3188.511719060" watchObservedRunningTime="2026-11-03 03:54:13.483603345 +0000 UTC m=+3188.514201883" Nov 03 03:54:19.373574 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:19.373543 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 03:54:25.378101 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.378065 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc"] Nov 03 03:54:25.383259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.383237 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.489935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.489892 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-286a8c-serving-cert\"" Nov 03 03:54:25.493008 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.492986 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc"] Nov 03 03:54:25.556814 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.556770 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h982c\" (UniqueName: \"kubernetes.io/projected/5294bb78-bd86-477b-b45e-d211e6cfd02f-kube-api-access-h982c\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.556990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.556836 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.556990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.556896 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294bb78-bd86-477b-b45e-d211e6cfd02f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.556990 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.556935 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.658889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.658790 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h982c\" (UniqueName: \"kubernetes.io/projected/5294bb78-bd86-477b-b45e-d211e6cfd02f-kube-api-access-h982c\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.658889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.658857 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.658889 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.658885 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294bb78-bd86-477b-b45e-d211e6cfd02f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.659189 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.658930 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.660171 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.660141 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294bb78-bd86-477b-b45e-d211e6cfd02f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.662079 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.662050 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.663630 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.663603 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.723429 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.723394 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h982c\" (UniqueName: \"kubernetes.io/projected/5294bb78-bd86-477b-b45e-d211e6cfd02f-kube-api-access-h982c\") pod \"maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:25.996720 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:25.996629 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:26.379295 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:26.379258 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc"] Nov 03 03:54:26.380239 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:54:26.380209 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5294bb78_bd86_477b_b45e_d211e6cfd02f.slice/crio-25dfab78d4eec992c6df56b1a80302cbb11154700a1813cde5a8a32c1462f9c2 WatchSource:0}: Error finding container 25dfab78d4eec992c6df56b1a80302cbb11154700a1813cde5a8a32c1462f9c2: Status 404 returned error can't find the container with id 25dfab78d4eec992c6df56b1a80302cbb11154700a1813cde5a8a32c1462f9c2 Nov 03 03:54:26.446089 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:26.446043 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" event={"ID":"5294bb78-bd86-477b-b45e-d211e6cfd02f","Type":"ContainerStarted","Data":"25dfab78d4eec992c6df56b1a80302cbb11154700a1813cde5a8a32c1462f9c2"} Nov 03 03:54:26.661991 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:26.661911 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8"] Nov 03 03:54:27.455045 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:27.455003 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" event={"ID":"5294bb78-bd86-477b-b45e-d211e6cfd02f","Type":"ContainerStarted","Data":"1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e"} Nov 03 03:54:27.455554 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:27.455222 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:27.577258 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:27.577204 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" podStartSLOduration=1.705773158 podStartE2EDuration="2.577191842s" totalImagesPullingTime="871.418684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:54:27.574036585 +0000 UTC m=+3202.604635125" watchObservedRunningTime="2026-11-03 03:54:27.577191842 +0000 UTC m=+3202.607790379" Nov 03 03:54:30.011522 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:54:30.011480 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:54:33.467940 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:33.467910 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 03:54:41.684560 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:41.684492 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" podUID="91c5a36e-446f-4b4d-9ca5-0bae41b084a7" containerName="main" containerID="cri-o://bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555" gracePeriod=45 Nov 03 03:54:41.958525 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:41.958496 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:54:42.060787 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.060754 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-home\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-home\") pod \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " Nov 03 03:54:42.060984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.060800 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-kube-api-access-4vz8k\" (UniqueName: \"kubernetes.io/projected/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-kube-api-access-4vz8k\") pod \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " Nov 03 03:54:42.060984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.060834 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-model-cache\") pod \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " Nov 03 03:54:42.060984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.060875 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-dshm\") pod \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " Nov 03 03:54:42.060984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.060931 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tls-certs\") pod \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " Nov 03 03:54:42.060984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.060985 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tmp-dir\") pod \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\" (UID: \"91c5a36e-446f-4b4d-9ca5-0bae41b084a7\") " Nov 03 03:54:42.061253 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.061094 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-home" pod "91c5a36e-446f-4b4d-9ca5-0bae41b084a7" (UID: "91c5a36e-446f-4b4d-9ca5-0bae41b084a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:54:42.061462 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.061411 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tmp-dir" pod "91c5a36e-446f-4b4d-9ca5-0bae41b084a7" (UID: "91c5a36e-446f-4b4d-9ca5-0bae41b084a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:54:42.061462 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.061428 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-model-cache" pod "91c5a36e-446f-4b4d-9ca5-0bae41b084a7" (UID: "91c5a36e-446f-4b4d-9ca5-0bae41b084a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:54:42.061647 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.061629 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:54:42.061702 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.061653 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:54:42.061702 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.061666 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:54:42.063732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.063704 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tls-certs" pod "91c5a36e-446f-4b4d-9ca5-0bae41b084a7" (UID: "91c5a36e-446f-4b4d-9ca5-0bae41b084a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:54:42.063877 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.063860 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-kube-api-access-4vz8k" pod "91c5a36e-446f-4b4d-9ca5-0bae41b084a7" (UID: "91c5a36e-446f-4b4d-9ca5-0bae41b084a7"). InnerVolumeSpecName "kube-api-access-4vz8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:54:42.063930 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.063886 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-dshm" pod "91c5a36e-446f-4b4d-9ca5-0bae41b084a7" (UID: "91c5a36e-446f-4b4d-9ca5-0bae41b084a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:54:42.162674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.162642 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:54:42.162674 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.162673 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:54:42.162866 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.162683 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vz8k\" (UniqueName: \"kubernetes.io/projected/91c5a36e-446f-4b4d-9ca5-0bae41b084a7-kube-api-access-4vz8k\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:54:42.547199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.547161 5117 generic.go:356] "Generic (PLEG): container finished" podID="91c5a36e-446f-4b4d-9ca5-0bae41b084a7" containerID="bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555" exitCode=0 Nov 03 03:54:42.547199 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.547198 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" event={"ID":"91c5a36e-446f-4b4d-9ca5-0bae41b084a7","Type":"ContainerDied","Data":"bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555"} Nov 03 03:54:42.547490 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.547207 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" Nov 03 03:54:42.547490 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.547227 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8" event={"ID":"91c5a36e-446f-4b4d-9ca5-0bae41b084a7","Type":"ContainerDied","Data":"7214e9c5de4c34b9851364a1b1a829f566b3c787090e10172649db98d2c6e013"} Nov 03 03:54:42.547490 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.547250 5117 scope.go:122] "RemoveContainer" containerID="bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555" Nov 03 03:54:42.557925 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.557908 5117 scope.go:122] "RemoveContainer" containerID="bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555" Nov 03 03:54:42.558193 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:54:42.558172 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555\": container with ID starting with bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555 not found: ID does not exist" containerID="bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555" Nov 03 03:54:42.558242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.558202 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555"} err="failed to get container status \"bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555\": rpc error: code = NotFound desc = could not find container \"bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555\": container with ID starting with bad96d3beca8d77562bde0ff90eefc32f85c2083d8bfcd303fd9f8e0d885e555 not found: ID does not exist" Nov 03 03:54:42.714628 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.714596 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8"] Nov 03 03:54:42.765391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:42.765358 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-158d23/explicit-model-158d23-kserve-657cb7cf89-pr4q8"] Nov 03 03:54:43.089760 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:54:43.089727 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c5a36e-446f-4b4d-9ca5-0bae41b084a7" path="/var/lib/kubelet/pods/91c5a36e-446f-4b4d-9ca5-0bae41b084a7/volumes" Nov 03 03:55:00.010482 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:55:00.010419 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:55:30.011123 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:55:30.011086 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:55:46.691229 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.691144 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b"] Nov 03 03:55:46.691879 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.691854 5117 cpu_manager.go:414] "containerMap: removing container" podUID="91c5a36e-446f-4b4d-9ca5-0bae41b084a7" containerName="main" Nov 03 03:55:46.691879 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.691877 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c5a36e-446f-4b4d-9ca5-0bae41b084a7" containerName="main" Nov 03 03:55:46.692078 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.692048 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c5a36e-446f-4b4d-9ca5-0bae41b084a7" containerName="main" Nov 03 03:55:46.695644 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.695625 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.759363 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.759333 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b"] Nov 03 03:55:46.781965 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.781930 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-home\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.782146 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.781976 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-model-cache\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.782146 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.782062 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tmp-dir\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.782146 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.782122 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdng8\" (UniqueName: \"kubernetes.io/projected/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-kube-api-access-zdng8\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.782330 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.782243 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tls-certs\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.782396 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.782364 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-dshm\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.804688 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.804663 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f3e02b\"/\"kube-root-ca.crt\"" Nov 03 03:55:46.804853 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.804836 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f3e02b\"/\"default-dockercfg-nbnlf\"" Nov 03 03:55:46.804915 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.804858 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f3e02b\"/\"openshift-service-ca.crt\"" Nov 03 03:55:46.804915 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.804904 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f3e02b\"/\"auth-test-model-f3e02b-kserve-self-signed-certs\"" Nov 03 03:55:46.883476 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.883421 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-home\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.883693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.883493 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-model-cache\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.883693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.883530 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tmp-dir\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.883693 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.883590 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdng8\" (UniqueName: \"kubernetes.io/projected/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-kube-api-access-zdng8\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.883899 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.883767 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tls-certs\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.883899 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.883877 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-dshm\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.884022 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.883896 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-home\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.884341 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.884318 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-model-cache\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.884718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.884663 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tmp-dir\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.887241 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.887215 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-dshm\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.887691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.887674 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tls-certs\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:46.946338 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:46.946264 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdng8\" (UniqueName: \"kubernetes.io/projected/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-kube-api-access-zdng8\") pod \"auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:47.008460 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:47.008409 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:47.397841 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:47.397807 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b"] Nov 03 03:55:47.398738 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:55:47.398699 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8f48b59_3ae9_40bd_b29b_f420b50f6c2a.slice/crio-2d566bc7cdd178b50f8599091a07dcaaa450ade1c992412d43a3a36bf1b19374 WatchSource:0}: Error finding container 2d566bc7cdd178b50f8599091a07dcaaa450ade1c992412d43a3a36bf1b19374: Status 404 returned error can't find the container with id 2d566bc7cdd178b50f8599091a07dcaaa450ade1c992412d43a3a36bf1b19374 Nov 03 03:55:47.979147 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:47.979107 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" event={"ID":"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a","Type":"ContainerStarted","Data":"86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad"} Nov 03 03:55:47.979147 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:47.979150 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" event={"ID":"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a","Type":"ContainerStarted","Data":"2d566bc7cdd178b50f8599091a07dcaaa450ade1c992412d43a3a36bf1b19374"} Nov 03 03:55:48.009496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:48.009465 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:48.101559 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:48.101510 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" podStartSLOduration=2.101489886 podStartE2EDuration="2.101489886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:55:48.098867911 +0000 UTC m=+3283.129466449" watchObservedRunningTime="2026-11-03 03:55:48.101489886 +0000 UTC m=+3283.132088423" Nov 03 03:55:57.008846 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:57.008799 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:57.022227 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:57.022195 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:55:57.049884 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:55:57.049852 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 03:56:00.014609 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:56:00.014558 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:56:05.864881 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.864843 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz"] Nov 03 03:56:05.868984 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.868967 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:05.905575 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.905542 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362725bd-ff9c-4608-9fc5-fe715d6a859d-tls-certs\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:05.905719 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.905631 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-home\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:05.905790 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.905769 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-tmp-dir\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:05.905831 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.905810 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-model-cache\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:05.905997 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.905947 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-dshm\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:05.906125 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.906103 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbn2\" (UniqueName: \"kubernetes.io/projected/362725bd-ff9c-4608-9fc5-fe715d6a859d-kube-api-access-ckbn2\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:05.975124 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.975086 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-10120d\"/\"openshift-service-ca.crt\"" Nov 03 03:56:05.975288 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.975173 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-10120d\"/\"kube-root-ca.crt\"" Nov 03 03:56:05.975288 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.975267 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-10120d\"/\"auth-test-model-10120d-kserve-self-signed-certs\"" Nov 03 03:56:05.975288 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.975265 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-10120d\"/\"default-dockercfg-xftq4\"" Nov 03 03:56:05.978493 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:05.978475 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz"] Nov 03 03:56:06.007152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007121 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbn2\" (UniqueName: \"kubernetes.io/projected/362725bd-ff9c-4608-9fc5-fe715d6a859d-kube-api-access-ckbn2\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007302 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007161 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362725bd-ff9c-4608-9fc5-fe715d6a859d-tls-certs\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007302 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007222 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-home\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007302 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007284 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-tmp-dir\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007304 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-model-cache\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007438 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007360 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-dshm\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007839 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007803 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-tmp-dir\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007943 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007848 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-model-cache\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.007943 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.007894 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-home\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.009802 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.009766 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-dshm\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.009924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.009908 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362725bd-ff9c-4608-9fc5-fe715d6a859d-tls-certs\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.070038 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.070008 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbn2\" (UniqueName: \"kubernetes.io/projected/362725bd-ff9c-4608-9fc5-fe715d6a859d-kube-api-access-ckbn2\") pod \"auth-test-model-10120d-kserve-6f75969b69-b9zzz\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.181391 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.181295 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:06.572512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:06.572482 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz"] Nov 03 03:56:06.572965 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:56:06.572935 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod362725bd_ff9c_4608_9fc5_fe715d6a859d.slice/crio-60e11240762b8b92e944da7c6a18107d7eb6689cd5235057e436976ef55de158 WatchSource:0}: Error finding container 60e11240762b8b92e944da7c6a18107d7eb6689cd5235057e436976ef55de158: Status 404 returned error can't find the container with id 60e11240762b8b92e944da7c6a18107d7eb6689cd5235057e436976ef55de158 Nov 03 03:56:07.109140 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.109104 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" event={"ID":"362725bd-ff9c-4608-9fc5-fe715d6a859d","Type":"ContainerStarted","Data":"dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3"} Nov 03 03:56:07.111522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.109148 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" event={"ID":"362725bd-ff9c-4608-9fc5-fe715d6a859d","Type":"ContainerStarted","Data":"60e11240762b8b92e944da7c6a18107d7eb6689cd5235057e436976ef55de158"} Nov 03 03:56:07.181736 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.181706 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:07.230744 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.230681 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" podStartSLOduration=2.230665208 podStartE2EDuration="2.230665208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:56:07.228287006 +0000 UTC m=+3302.258885546" watchObservedRunningTime="2026-11-03 03:56:07.230665208 +0000 UTC m=+3302.261263746" Nov 03 03:56:07.615973 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.615939 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 03:56:07.622205 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.622178 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:56:07.624183 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.624159 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 03:56:07.629994 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:07.629972 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 03:56:16.182182 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:16.182149 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:16.194850 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:16.194826 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:17.186899 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:17.186868 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 03:56:23.367466 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.367411 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d"] Nov 03 03:56:23.372914 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.372892 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.479862 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.479818 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"openshift-service-ca.crt\"" Nov 03 03:56:23.480056 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.479865 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"kube-root-ca.crt\"" Nov 03 03:56:23.480056 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.479943 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"default-dockercfg-c5z2n\"" Nov 03 03:56:23.480191 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.480087 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"auth-test-model-66d902-kserve-self-signed-certs\"" Nov 03 03:56:23.483328 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.483307 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d"] Nov 03 03:56:23.524512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.524472 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-home\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.524779 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.524528 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-model-cache\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.524779 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.524558 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gls49\" (UniqueName: \"kubernetes.io/projected/648b8c29-69fd-4406-9b00-84ce1e72577e-kube-api-access-gls49\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.524779 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.524597 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648b8c29-69fd-4406-9b00-84ce1e72577e-tls-certs\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.524995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.524866 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-dshm\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.524995 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.524899 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-tmp-dir\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.625893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.625792 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-dshm\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.625893 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.625832 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-tmp-dir\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.626202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.625979 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-home\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.626202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.626026 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-model-cache\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.626202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.626053 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gls49\" (UniqueName: \"kubernetes.io/projected/648b8c29-69fd-4406-9b00-84ce1e72577e-kube-api-access-gls49\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.626202 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.626091 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648b8c29-69fd-4406-9b00-84ce1e72577e-tls-certs\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.626424 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.626227 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-tmp-dir\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.626531 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.626463 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-home\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.626625 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.626605 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-model-cache\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.629050 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.629011 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-dshm\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.629226 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.629206 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648b8c29-69fd-4406-9b00-84ce1e72577e-tls-certs\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.694003 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.693968 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gls49\" (UniqueName: \"kubernetes.io/projected/648b8c29-69fd-4406-9b00-84ce1e72577e-kube-api-access-gls49\") pod \"auth-test-model-66d902-kserve-84ccc54dd9-sw42d\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:23.986385 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:23.986303 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:24.367459 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:24.367418 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d"] Nov 03 03:56:24.367598 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:56:24.367546 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod648b8c29_69fd_4406_9b00_84ce1e72577e.slice/crio-f222ad91587cd3904c9ed5b68bad8af16da93e4a83b542bc085b7a6006e839a7 WatchSource:0}: Error finding container f222ad91587cd3904c9ed5b68bad8af16da93e4a83b542bc085b7a6006e839a7: Status 404 returned error can't find the container with id f222ad91587cd3904c9ed5b68bad8af16da93e4a83b542bc085b7a6006e839a7 Nov 03 03:56:25.226543 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:25.226498 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" event={"ID":"648b8c29-69fd-4406-9b00-84ce1e72577e","Type":"ContainerStarted","Data":"7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372"} Nov 03 03:56:25.226714 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:25.226556 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" event={"ID":"648b8c29-69fd-4406-9b00-84ce1e72577e","Type":"ContainerStarted","Data":"f222ad91587cd3904c9ed5b68bad8af16da93e4a83b542bc085b7a6006e839a7"} Nov 03 03:56:25.347513 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:25.347427 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" podStartSLOduration=2.347414365 podStartE2EDuration="2.347414365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:56:25.34532124 +0000 UTC m=+3320.375919781" watchObservedRunningTime="2026-11-03 03:56:25.347414365 +0000 UTC m=+3320.378012904" Nov 03 03:56:25.987323 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:25.987288 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:30.012655 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:56:30.012612 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:56:33.986631 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:33.986593 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:33.998895 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:33.998874 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:34.297370 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:34.297287 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 03:56:42.977851 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:42.977812 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn"] Nov 03 03:56:42.978283 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:42.978069 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" containerID="cri-o://ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a" gracePeriod=30 Nov 03 03:56:43.762213 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:43.762170 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-kjzbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:56:43.762399 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:43.762276 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:56:48.256142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.256064 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:56:48.322980 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.322942 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-podinfo\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323136 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.322994 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-credential-socket\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323178 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323159 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-kube-api-access-b65jf\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-kube-api-access-b65jf\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323223 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323209 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-envoy\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323378 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323352 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-token\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323378 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323365 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-credential-socket" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:48.323610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323440 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-socket\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323543 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istiod-ca-cert\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323610 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323576 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-certs\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.323801 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.323635 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-data\") pod \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\" (UID: \"c861579f-cd2c-4c46-9e25-ea2e49913e8e\") " Nov 03 03:56:48.324212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324030 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-certs" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324216 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-data" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324285 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-socket" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324305 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istiod-ca-cert" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324473 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324488 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324498 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324506 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.324612 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.324520 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.325653 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.325630 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-podinfo" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:56:48.325907 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.325884 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-token" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:48.326061 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.326043 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-kube-api-access-b65jf" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "kube-api-access-b65jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:48.326153 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.326064 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-envoy" pod "c861579f-cd2c-4c46-9e25-ea2e49913e8e" (UID: "c861579f-cd2c-4c46-9e25-ea2e49913e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:48.393255 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.393223 5117 generic.go:356] "Generic (PLEG): container finished" podID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerID="ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a" exitCode=0 Nov 03 03:56:48.393408 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.393269 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" Nov 03 03:56:48.393408 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.393310 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" event={"ID":"c861579f-cd2c-4c46-9e25-ea2e49913e8e","Type":"ContainerDied","Data":"ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a"} Nov 03 03:56:48.393408 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.393362 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn" event={"ID":"c861579f-cd2c-4c46-9e25-ea2e49913e8e","Type":"ContainerDied","Data":"0e64b87d4c341f792bca4a6c3342090f2388342b58823ec781ffbfc39003a17e"} Nov 03 03:56:48.393408 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.393390 5117 scope.go:122] "RemoveContainer" containerID="ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a" Nov 03 03:56:48.404683 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.404664 5117 scope.go:122] "RemoveContainer" containerID="ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a" Nov 03 03:56:48.404956 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:56:48.404939 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a\": container with ID starting with ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a not found: ID does not exist" containerID="ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a" Nov 03 03:56:48.405002 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.404966 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a"} err="failed to get container status \"ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a\": rpc error: code = NotFound desc = could not find container \"ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a\": container with ID starting with ca87b368a74d5c57829bd8a2ad1f569f2ce0a599d211670f7e33aba82ff35a1a not found: ID does not exist" Nov 03 03:56:48.425961 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.425930 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.425961 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.425953 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b65jf\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-kube-api-access-b65jf\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.426103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.425964 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.426103 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.425974 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c861579f-cd2c-4c46-9e25-ea2e49913e8e-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:48.486935 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.486894 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z"] Nov 03 03:56:48.487433 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.487420 5117 cpu_manager.go:414] "containerMap: removing container" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" Nov 03 03:56:48.487537 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.487433 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" Nov 03 03:56:48.487588 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.487571 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" containerName="istio-proxy" Nov 03 03:56:48.492379 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.492359 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.546434 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.546209 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-efa9bf\"/\"kube-root-ca.crt\"" Nov 03 03:56:48.546434 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.546393 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-efa9bf\"/\"openshift-service-ca.crt\"" Nov 03 03:56:48.546817 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.546787 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-efa9bf\"/\"default-dockercfg-5r8s7\"" Nov 03 03:56:48.546962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.546825 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn"] Nov 03 03:56:48.546962 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.546831 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-efa9bf\"/\"auth-test-model-efa9bf-kserve-self-signed-certs\"" Nov 03 03:56:48.551108 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.551079 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z"] Nov 03 03:56:48.597396 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.597365 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-kjzbn"] Nov 03 03:56:48.629242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.629204 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-tmp-dir\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.629496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.629363 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-dshm\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.629496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.629396 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75hgz\" (UniqueName: \"kubernetes.io/projected/1449b617-7602-4f4b-a0d4-4561158bd3df-kube-api-access-75hgz\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.629637 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.629506 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-home\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.629691 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.629633 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449b617-7602-4f4b-a0d4-4561158bd3df-tls-certs\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.629743 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.629697 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-model-cache\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731325 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731292 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-model-cache\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731358 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-tmp-dir\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731583 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731519 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-dshm\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731583 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731566 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75hgz\" (UniqueName: \"kubernetes.io/projected/1449b617-7602-4f4b-a0d4-4561158bd3df-kube-api-access-75hgz\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731692 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731624 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-home\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731753 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731739 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449b617-7602-4f4b-a0d4-4561158bd3df-tls-certs\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731778 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-tmp-dir\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.731812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.731792 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-model-cache\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.732082 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.732053 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-home\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.733886 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.733864 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-dshm\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.734284 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.734264 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449b617-7602-4f4b-a0d4-4561158bd3df-tls-certs\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.794365 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.794333 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75hgz\" (UniqueName: \"kubernetes.io/projected/1449b617-7602-4f4b-a0d4-4561158bd3df-kube-api-access-75hgz\") pod \"auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:48.808242 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:48.808151 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:49.092420 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:49.092388 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c861579f-cd2c-4c46-9e25-ea2e49913e8e" path="/var/lib/kubelet/pods/c861579f-cd2c-4c46-9e25-ea2e49913e8e/volumes" Nov 03 03:56:49.202061 ip-10-0-138-120 kubenswrapper[5117]: W1103 03:56:49.202031 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1449b617_7602_4f4b_a0d4_4561158bd3df.slice/crio-881c32b866dc182c8276b64c32bb0ffdf248ba043f3f17281a7d7db3f22a6b96 WatchSource:0}: Error finding container 881c32b866dc182c8276b64c32bb0ffdf248ba043f3f17281a7d7db3f22a6b96: Status 404 returned error can't find the container with id 881c32b866dc182c8276b64c32bb0ffdf248ba043f3f17281a7d7db3f22a6b96 Nov 03 03:56:49.203152 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:49.203122 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z"] Nov 03 03:56:49.401527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:49.401362 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" event={"ID":"1449b617-7602-4f4b-a0d4-4561158bd3df","Type":"ContainerStarted","Data":"5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261"} Nov 03 03:56:49.401527 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:49.401434 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" event={"ID":"1449b617-7602-4f4b-a0d4-4561158bd3df","Type":"ContainerStarted","Data":"881c32b866dc182c8276b64c32bb0ffdf248ba043f3f17281a7d7db3f22a6b96"} Nov 03 03:56:49.521956 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:49.521903 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" podStartSLOduration=1.5218880700000001 podStartE2EDuration="1.52188807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:56:49.520114835 +0000 UTC m=+3344.550713353" watchObservedRunningTime="2026-11-03 03:56:49.52188807 +0000 UTC m=+3344.552486607" Nov 03 03:56:49.808895 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:49.808816 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:53.010651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:53.010613 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt"] Nov 03 03:56:53.011063 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:53.010857 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" containerID="cri-o://3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868" gracePeriod=30 Nov 03 03:56:58.286884 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.286814 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:56:58.453562 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453520 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-socket\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.453732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453584 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15074a77-6cf3-4d5b-b9f5-e275361bee20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15074a77-6cf3-4d5b-b9f5-e275361bee20-istiod-ca-cert\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.453732 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453659 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-envoy\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.453840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453731 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-kube-api-access-4hwcr\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-kube-api-access-4hwcr\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.453840 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453777 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-token\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-token\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.453924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453838 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-socket" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.453924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453906 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-credential-socket\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.454008 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453951 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-podinfo\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.454008 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453983 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-data\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.454008 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.453980 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15074a77-6cf3-4d5b-b9f5-e275361bee20-istiod-ca-cert" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:56:58.454179 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454026 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-certs\") pod \"15074a77-6cf3-4d5b-b9f5-e275361bee20\" (UID: \"15074a77-6cf3-4d5b-b9f5-e275361bee20\") " Nov 03 03:56:58.454179 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454123 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-credential-socket" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.454295 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454254 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-data" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.454398 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454364 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-certs" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.454892 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454864 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.455024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454897 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.455024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454912 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.455024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454926 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.455024 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.454941 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15074a77-6cf3-4d5b-b9f5-e275361bee20-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.456281 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.456258 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-envoy" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:56:58.456367 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.456349 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-token" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.456740 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.456718 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-podinfo" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:56:58.456863 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.456802 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-kube-api-access-4hwcr" pod "15074a77-6cf3-4d5b-b9f5-e275361bee20" (UID: "15074a77-6cf3-4d5b-b9f5-e275361bee20"). InnerVolumeSpecName "kube-api-access-4hwcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:56:58.471139 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.471110 5117 generic.go:356] "Generic (PLEG): container finished" podID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerID="3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868" exitCode=0 Nov 03 03:56:58.471277 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.471249 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" event={"ID":"15074a77-6cf3-4d5b-b9f5-e275361bee20","Type":"ContainerDied","Data":"3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868"} Nov 03 03:56:58.471345 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.471335 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" event={"ID":"15074a77-6cf3-4d5b-b9f5-e275361bee20","Type":"ContainerDied","Data":"5e57ffe0d167726d31c8067907f57ea17f4eaa438fead995774b051d11650663"} Nov 03 03:56:58.471402 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.471369 5117 scope.go:122] "RemoveContainer" containerID="3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868" Nov 03 03:56:58.472495 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.471584 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt" Nov 03 03:56:58.493297 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.493277 5117 scope.go:122] "RemoveContainer" containerID="3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868" Nov 03 03:56:58.493620 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:56:58.493597 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868\": container with ID starting with 3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868 not found: ID does not exist" containerID="3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868" Nov 03 03:56:58.493710 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.493627 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868"} err="failed to get container status \"3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868\": rpc error: code = NotFound desc = could not find container \"3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868\": container with ID starting with 3ca9dddbfcc38d19cf84603eae17db201bc9ef5396d5c1e686781e1135730868 not found: ID does not exist" Nov 03 03:56:58.556733 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.556657 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.556733 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.556684 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.556733 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.556695 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwcr\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-kube-api-access-4hwcr\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.556733 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.556706 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15074a77-6cf3-4d5b-b9f5-e275361bee20-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:56:58.642576 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.642542 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt"] Nov 03 03:56:58.646960 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.646930 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dghxt"] Nov 03 03:56:58.808524 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.808383 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:58.821752 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:58.821729 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:56:59.091145 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:59.091118 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" path="/var/lib/kubelet/pods/15074a77-6cf3-4d5b-b9f5-e275361bee20/volumes" Nov 03 03:56:59.490017 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:56:59.489937 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 03:57:00.013147 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:57:00.013109 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:57:08.294843 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.294802 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n"] Nov 03 03:57:08.295331 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.295056 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" podUID="70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" containerName="maas-api" containerID="cri-o://1c8b24cfd05a5da1c28dfe92f6b828274a0c0b99152b429d1010159a7f451ce6" gracePeriod=30 Nov 03 03:57:08.566956 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.566919 5117 generic.go:356] "Generic (PLEG): container finished" podID="70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" containerID="1c8b24cfd05a5da1c28dfe92f6b828274a0c0b99152b429d1010159a7f451ce6" exitCode=0 Nov 03 03:57:08.567371 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.567049 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" event={"ID":"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e","Type":"ContainerDied","Data":"1c8b24cfd05a5da1c28dfe92f6b828274a0c0b99152b429d1010159a7f451ce6"} Nov 03 03:57:08.604298 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.604267 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:57:08.672685 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.672647 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-kube-api-access-fwcmm\" (UniqueName: \"kubernetes.io/projected/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-kube-api-access-fwcmm\") pod \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " Nov 03 03:57:08.672900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.672707 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-metrics-certs\") pod \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " Nov 03 03:57:08.672900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.672727 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-openshift-service-ca\") pod \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " Nov 03 03:57:08.672900 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.672879 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls\") pod \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\" (UID: \"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e\") " Nov 03 03:57:08.673233 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.673205 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-openshift-service-ca" pod "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" (UID: "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:57:08.673385 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.673369 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:08.675105 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.675077 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-metrics-certs" pod "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" (UID: "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:57:08.675204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.675101 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls" pod "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" (UID: "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:57:08.675204 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.675108 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-kube-api-access-fwcmm" pod "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" (UID: "70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e"). InnerVolumeSpecName "kube-api-access-fwcmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:57:08.774417 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.774371 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:08.774417 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.774413 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwcmm\" (UniqueName: \"kubernetes.io/projected/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-kube-api-access-fwcmm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:08.774663 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:08.774432 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:57:09.575350 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:09.575316 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" Nov 03 03:57:09.575350 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:09.575336 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n" event={"ID":"70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e","Type":"ContainerDied","Data":"c3de9efbc59e2721925f38b641551b9938a6d513663cbf5ad8cb26c0d92d517b"} Nov 03 03:57:09.575951 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:09.575406 5117 scope.go:122] "RemoveContainer" containerID="1c8b24cfd05a5da1c28dfe92f6b828274a0c0b99152b429d1010159a7f451ce6" Nov 03 03:57:09.754618 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:09.754583 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n"] Nov 03 03:57:09.807190 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:09.807157 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-158d23-865fb6567c-9zs7n"] Nov 03 03:57:11.089812 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:57:11.089777 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" path="/var/lib/kubelet/pods/70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e/volumes" Nov 03 03:57:30.011492 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:57:30.011435 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:58:00.011881 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:58:00.011843 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:58:04.527360 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:04.527319 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq"] Nov 03 03:58:04.527927 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:04.527601 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" containerID="cri-o://f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67" gracePeriod=30 Nov 03 03:58:09.131659 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.131621 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 03:58:09.132050 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.131687 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 03:58:09.790389 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.790356 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:58:09.912738 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.912695 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-kube-api-access-sh64b\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-kube-api-access-sh64b\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.912924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.912763 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-podinfo\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.912924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.912828 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-envoy\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.912924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.912850 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-credential-socket\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.912924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.912889 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-socket\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.912924 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.912925 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istiod-ca-cert\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.913192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913018 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-token\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-token\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.913192 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913173 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-certs\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.913308 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913170 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-socket" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:09.913308 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913197 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-credential-socket" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:09.913424 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913336 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-certs" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:09.913522 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913499 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istiod-ca-cert" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:58:09.913675 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913626 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-data" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:09.914001 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.913977 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-data\") pod \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\" (UID: \"3660a45d-bddf-4804-a5af-d6c9050bfe1f\") " Nov 03 03:58:09.914584 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.914563 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:09.914634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.914590 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:09.914634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.914606 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:09.914634 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.914623 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:09.914745 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.914638 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:09.915269 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.915247 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-envoy" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:09.915461 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.915424 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-token" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:09.915601 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.915581 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-podinfo" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:58:09.915656 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.915597 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-kube-api-access-sh64b" pod "3660a45d-bddf-4804-a5af-d6c9050bfe1f" (UID: "3660a45d-bddf-4804-a5af-d6c9050bfe1f"). InnerVolumeSpecName "kube-api-access-sh64b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:09.988252 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.988220 5117 generic.go:356] "Generic (PLEG): container finished" podID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerID="f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67" exitCode=0 Nov 03 03:58:09.988430 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.988275 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" Nov 03 03:58:09.988430 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.988313 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" event={"ID":"3660a45d-bddf-4804-a5af-d6c9050bfe1f","Type":"ContainerDied","Data":"f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67"} Nov 03 03:58:09.988430 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.988368 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq" event={"ID":"3660a45d-bddf-4804-a5af-d6c9050bfe1f","Type":"ContainerDied","Data":"2b6e0dd6a0575fd15a94ab178fb2cdf499907d3742d1e8db8f15877837d59295"} Nov 03 03:58:09.988430 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:09.988392 5117 scope.go:122] "RemoveContainer" containerID="f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67" Nov 03 03:58:10.000212 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.000195 5117 scope.go:122] "RemoveContainer" containerID="f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67" Nov 03 03:58:10.000553 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:58:10.000530 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67\": container with ID starting with f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67 not found: ID does not exist" containerID="f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67" Nov 03 03:58:10.000620 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.000567 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67"} err="failed to get container status \"f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67\": rpc error: code = NotFound desc = could not find container \"f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67\": container with ID starting with f3839526459df524434c3b349b1662218a610bce7d1d6084d19d96612ef1eb67 not found: ID does not exist" Nov 03 03:58:10.015506 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.015481 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:10.015506 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.015502 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:10.015651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.015513 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh64b\" (UniqueName: \"kubernetes.io/projected/3660a45d-bddf-4804-a5af-d6c9050bfe1f-kube-api-access-sh64b\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:10.015651 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.015523 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3660a45d-bddf-4804-a5af-d6c9050bfe1f-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:10.156729 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.156692 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq"] Nov 03 03:58:10.162354 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:10.162327 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-4ncsq"] Nov 03 03:58:11.090696 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:11.090661 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" path="/var/lib/kubelet/pods/3660a45d-bddf-4804-a5af-d6c9050bfe1f/volumes" Nov 03 03:58:23.727496 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:23.727429 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c"] Nov 03 03:58:23.727911 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:23.727689 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" containerID="cri-o://43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee" gracePeriod=30 Nov 03 03:58:28.991520 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:28.991426 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:58:29.046461 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046417 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-socket\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046611 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046576 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-podinfo\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046662 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046647 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-kube-api-access-fmfd8\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-kube-api-access-fmfd8\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046718 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046693 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-socket" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:29.046785 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046747 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-data\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046843 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046789 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istiod-ca-cert\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046848 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-credential-socket\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046898 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046880 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-envoy\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046925 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-token\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-token\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.046999 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.046992 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-certs\") pod \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\" (UID: \"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b\") " Nov 03 03:58:29.047111 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.047039 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-data" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:29.047410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.047252 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-credential-socket" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:29.047410 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.047284 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istiod-ca-cert" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:58:29.047676 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.047458 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-certs" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:29.048142 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.048115 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.048259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.048149 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.048259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.048166 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.048259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.048180 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.048259 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.048195 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.049512 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.049471 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-kube-api-access-fmfd8" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "kube-api-access-fmfd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:29.049703 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.049680 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-token" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:58:29.049786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.049765 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-podinfo" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 03:58:29.049786 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.049771 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-envoy" pod "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" (UID: "50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 03:58:29.120806 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.120776 5117 generic.go:356] "Generic (PLEG): container finished" podID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerID="43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee" exitCode=0 Nov 03 03:58:29.120937 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.120817 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" Nov 03 03:58:29.120937 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.120859 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" event={"ID":"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b","Type":"ContainerDied","Data":"43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee"} Nov 03 03:58:29.120937 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.120907 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c" event={"ID":"50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b","Type":"ContainerDied","Data":"4c078706eb7ae3806d7a39baf4f71d9b824f8edcfb9a52d353ca9dcd69a97234"} Nov 03 03:58:29.121097 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.120938 5117 scope.go:122] "RemoveContainer" containerID="43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee" Nov 03 03:58:29.131284 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.131263 5117 scope.go:122] "RemoveContainer" containerID="43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee" Nov 03 03:58:29.131563 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:58:29.131545 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee\": container with ID starting with 43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee not found: ID does not exist" containerID="43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee" Nov 03 03:58:29.131645 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.131571 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee"} err="failed to get container status \"43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee\": rpc error: code = NotFound desc = could not find container \"43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee\": container with ID starting with 43797f33605480ba88f6939fb44664c1011cfb2165fd4820b47603675653c0ee not found: ID does not exist" Nov 03 03:58:29.149485 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.149457 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.149602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.149492 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.149602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.149510 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.149602 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.149525 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmfd8\" (UniqueName: \"kubernetes.io/projected/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b-kube-api-access-fmfd8\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 03:58:29.281373 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.281284 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c"] Nov 03 03:58:29.285172 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:29.285144 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-98c4c"] Nov 03 03:58:30.010838 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:58:30.010803 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:58:31.090322 ip-10-0-138-120 kubenswrapper[5117]: I1103 03:58:31.090288 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" path="/var/lib/kubelet/pods/50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b/volumes" Nov 03 03:59:00.013356 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:59:00.013315 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 03:59:30.011525 ip-10-0-138-120 kubenswrapper[5117]: E1103 03:59:30.011484 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:00:00.012070 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:00:00.012031 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:00:20.576775 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:20.576739 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg"] Nov 03 04:00:20.577217 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:20.576953 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" containerID="cri-o://449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581" gracePeriod=30 Nov 03 04:00:24.235275 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:24.235243 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:00:24.235705 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:24.235292 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:00:24.666611 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:24.666573 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b"] Nov 03 04:00:24.666833 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:24.666808 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" containerID="cri-o://e4325def31f36a42e3d1f962fec234896870802181a7f3cf2b592d4fa9d8c1a2" gracePeriod=30 Nov 03 04:00:25.837738 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.837710 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 04:00:25.858206 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.858135 5117 generic.go:356] "Generic (PLEG): container finished" podID="96310c43-9954-4cfe-907a-df1272d49305" containerID="449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581" exitCode=0 Nov 03 04:00:25.858206 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.858180 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" Nov 03 04:00:25.858409 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.858183 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" event={"ID":"96310c43-9954-4cfe-907a-df1272d49305","Type":"ContainerDied","Data":"449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581"} Nov 03 04:00:25.858409 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.858325 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg" event={"ID":"96310c43-9954-4cfe-907a-df1272d49305","Type":"ContainerDied","Data":"5e8968c5ce34c6fe0f35b207995e92be6851d7649d2c7480fc1f33d44faef7f5"} Nov 03 04:00:25.858409 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.858364 5117 scope.go:122] "RemoveContainer" containerID="449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581" Nov 03 04:00:25.868812 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.868796 5117 scope.go:122] "RemoveContainer" containerID="449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581" Nov 03 04:00:25.869053 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:00:25.869033 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581\": container with ID starting with 449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581 not found: ID does not exist" containerID="449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581" Nov 03 04:00:25.869117 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.869064 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581"} err="failed to get container status \"449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581\": rpc error: code = NotFound desc = could not find container \"449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581\": container with ID starting with 449eae5ddea9faeef183c9c13f1ff6609ed4eea2e4cf646d5de5542502b2e581 not found: ID does not exist" Nov 03 04:00:25.945104 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945071 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-kube-api-access-kqkr4\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-kube-api-access-kqkr4\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945306 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945150 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-credential-socket\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945306 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945182 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-istio-token\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-istio-token\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945306 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945224 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-data\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945518 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945327 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-certs\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945518 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945360 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96310c43-9954-4cfe-907a-df1272d49305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96310c43-9954-4cfe-907a-df1272d49305-istio-podinfo\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945518 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945397 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96310c43-9954-4cfe-907a-df1272d49305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96310c43-9954-4cfe-907a-df1272d49305-istiod-ca-cert\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945518 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945481 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-socket\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945741 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945522 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-credential-socket" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:25.945741 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945565 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-envoy\") pod \"96310c43-9954-4cfe-907a-df1272d49305\" (UID: \"96310c43-9954-4cfe-907a-df1272d49305\") " Nov 03 04:00:25.945741 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945566 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-data" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:25.945741 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945633 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-certs" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:25.946115 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945783 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-socket" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:25.946115 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.945999 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:25.946115 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.946016 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:25.946115 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.946026 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:25.946115 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.946039 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:25.946385 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.946148 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96310c43-9954-4cfe-907a-df1272d49305-istiod-ca-cert" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:25.947714 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.947684 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-kube-api-access-kqkr4" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "kube-api-access-kqkr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:25.947832 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.947754 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-istio-token" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:25.947832 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.947781 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96310c43-9954-4cfe-907a-df1272d49305-istio-podinfo" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:00:25.947940 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:25.947923 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-envoy" pod "96310c43-9954-4cfe-907a-df1272d49305" (UID: "96310c43-9954-4cfe-907a-df1272d49305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:26.047254 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:26.047221 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:26.047254 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:26.047254 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96310c43-9954-4cfe-907a-df1272d49305-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:26.047482 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:26.047270 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96310c43-9954-4cfe-907a-df1272d49305-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:26.047482 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:26.047284 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96310c43-9954-4cfe-907a-df1272d49305-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:26.047482 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:26.047298 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkr4\" (UniqueName: \"kubernetes.io/projected/96310c43-9954-4cfe-907a-df1272d49305-kube-api-access-kqkr4\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:26.282865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:26.282828 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg"] Nov 03 04:00:26.290136 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:26.290098 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8kfqg"] Nov 03 04:00:27.091460 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:27.091415 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96310c43-9954-4cfe-907a-df1272d49305" path="/var/lib/kubelet/pods/96310c43-9954-4cfe-907a-df1272d49305/volumes" Nov 03 04:00:29.889224 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:29.889188 5117 generic.go:356] "Generic (PLEG): container finished" podID="604d0d93-538e-498a-a3dc-b819b663eb30" containerID="e4325def31f36a42e3d1f962fec234896870802181a7f3cf2b592d4fa9d8c1a2" exitCode=0 Nov 03 04:00:29.889662 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:29.889232 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" event={"ID":"604d0d93-538e-498a-a3dc-b819b663eb30","Type":"ContainerDied","Data":"e4325def31f36a42e3d1f962fec234896870802181a7f3cf2b592d4fa9d8c1a2"} Nov 03 04:00:29.945350 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:29.945296 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 04:00:30.012130 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:00:30.012090 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095176 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-istio-token\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-istio-token\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095270 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-certs\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095316 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-envoy\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095353 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-kube-api-access-w7f2r\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-kube-api-access-w7f2r\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095380 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-data\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095425 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-credential-socket\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095486 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-socket\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095519 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/604d0d93-538e-498a-a3dc-b819b663eb30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d0d93-538e-498a-a3dc-b819b663eb30-istio-podinfo\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095547 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604d0d93-538e-498a-a3dc-b819b663eb30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d0d93-538e-498a-a3dc-b819b663eb30-istiod-ca-cert\") pod \"604d0d93-538e-498a-a3dc-b819b663eb30\" (UID: \"604d0d93-538e-498a-a3dc-b819b663eb30\") " Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095659 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-certs" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.095884 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-socket" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.096012 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-data" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:30.096200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.096027 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-credential-socket" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:30.097085 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.096271 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604d0d93-538e-498a-a3dc-b819b663eb30-istiod-ca-cert" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:00:30.097283 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.097260 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.097436 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.097421 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.097572 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.097557 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.097718 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.097704 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.097847 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.097832 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604d0d93-538e-498a-a3dc-b819b663eb30-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.098625 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.098471 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-envoy" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:00:30.098625 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.098597 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-kube-api-access-w7f2r" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "kube-api-access-w7f2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:30.099755 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.099730 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-istio-token" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:00:30.100136 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.100108 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/604d0d93-538e-498a-a3dc-b819b663eb30-istio-podinfo" pod "604d0d93-538e-498a-a3dc-b819b663eb30" (UID: "604d0d93-538e-498a-a3dc-b819b663eb30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:00:30.198802 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.198727 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.198802 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.198752 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604d0d93-538e-498a-a3dc-b819b663eb30-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.198802 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.198762 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7f2r\" (UniqueName: \"kubernetes.io/projected/604d0d93-538e-498a-a3dc-b819b663eb30-kube-api-access-w7f2r\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.198802 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.198772 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604d0d93-538e-498a-a3dc-b819b663eb30-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:00:30.897193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.897164 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" Nov 03 04:00:30.897629 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.897160 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b" event={"ID":"604d0d93-538e-498a-a3dc-b819b663eb30","Type":"ContainerDied","Data":"4442063e88744706761c33e1c4038aa7534094875433f6c9c3a126a0fb753831"} Nov 03 04:00:30.897629 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:30.897289 5117 scope.go:122] "RemoveContainer" containerID="e4325def31f36a42e3d1f962fec234896870802181a7f3cf2b592d4fa9d8c1a2" Nov 03 04:00:31.066726 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:31.066695 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b"] Nov 03 04:00:31.117047 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:31.117009 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-q4g6b"] Nov 03 04:00:33.089652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:00:33.089619 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" path="/var/lib/kubelet/pods/604d0d93-538e-498a-a3dc-b819b663eb30/volumes" Nov 03 04:01:00.012260 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:01:00.012214 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:01:07.660894 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:01:07.660857 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:01:07.665956 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:01:07.665930 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:01:07.669606 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:01:07.669586 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:01:07.674759 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:01:07.674740 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:01:30.010115 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:01:30.010077 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:02:00.011137 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:02:00.011097 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:02:30.011347 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:02:30.011297 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:02:36.384409 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:36.384375 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz"] Nov 03 04:02:36.657010 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:36.656926 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg"] Nov 03 04:02:36.657171 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:36.657127 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" containerID="cri-o://949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21" gracePeriod=30 Nov 03 04:02:37.327083 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:37.327045 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b"] Nov 03 04:02:38.311751 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:38.311710 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d"] Nov 03 04:02:38.777022 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:38.776942 5117 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:02:38.777167 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:38.777020 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:02:41.943152 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.943087 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 04:02:41.979294 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979266 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-podinfo\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979495 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979324 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-token\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-token\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979495 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979353 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istiod-ca-cert\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979495 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979406 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-envoy\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979688 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979523 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-kube-api-access-djrjv\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-kube-api-access-djrjv\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979688 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979574 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-certs\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979688 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979630 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-data\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979688 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979668 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-socket\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979917 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979706 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-credential-socket\") pod \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\" (UID: \"b0cad53d-ab1f-47c3-91e4-97d8e432f889\") " Nov 03 04:02:41.979975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979907 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-certs" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:41.979975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.979920 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-data" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:41.980220 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.980171 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-credential-socket" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:41.980405 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.980323 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-socket" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:41.980495 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.980425 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:41.980495 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.980478 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:41.980621 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.980559 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istiod-ca-cert" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:02:41.982127 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.982096 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-podinfo" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:02:41.982127 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.982105 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-token" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:41.982301 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.982161 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-envoy" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:41.982370 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:41.982311 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-kube-api-access-djrjv" pod "b0cad53d-ab1f-47c3-91e4-97d8e432f889" (UID: "b0cad53d-ab1f-47c3-91e4-97d8e432f889"). InnerVolumeSpecName "kube-api-access-djrjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:42.081362 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.081326 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:42.081362 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.081363 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:42.081615 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.081380 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:42.081615 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.081393 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:42.081615 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.081409 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:42.081615 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.081422 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0cad53d-ab1f-47c3-91e4-97d8e432f889-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:42.081615 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.081472 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djrjv\" (UniqueName: \"kubernetes.io/projected/b0cad53d-ab1f-47c3-91e4-97d8e432f889-kube-api-access-djrjv\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:42.718819 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.718784 5117 generic.go:356] "Generic (PLEG): container finished" podID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerID="949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21" exitCode=0 Nov 03 04:02:42.719036 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.718838 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" Nov 03 04:02:42.719036 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.718868 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" event={"ID":"b0cad53d-ab1f-47c3-91e4-97d8e432f889","Type":"ContainerDied","Data":"949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21"} Nov 03 04:02:42.719036 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.718932 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg" event={"ID":"b0cad53d-ab1f-47c3-91e4-97d8e432f889","Type":"ContainerDied","Data":"babe898182e7671f42c63e52414b5980f13482acb460bb65b6df904eb584adb4"} Nov 03 04:02:42.719036 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.718962 5117 scope.go:122] "RemoveContainer" containerID="949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21" Nov 03 04:02:42.731645 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.731622 5117 scope.go:122] "RemoveContainer" containerID="949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21" Nov 03 04:02:42.731935 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:02:42.731905 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21\": container with ID starting with 949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21 not found: ID does not exist" containerID="949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21" Nov 03 04:02:42.732007 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.731949 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21"} err="failed to get container status \"949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21\": rpc error: code = NotFound desc = could not find container \"949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21\": container with ID starting with 949a95d61c6aca357db2c15123e313c4e05afe893274a9cf1de04af841882f21 not found: ID does not exist" Nov 03 04:02:42.891642 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.891605 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg"] Nov 03 04:02:42.938586 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:42.938552 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-rd5rg"] Nov 03 04:02:43.090036 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:43.090001 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" path="/var/lib/kubelet/pods/b0cad53d-ab1f-47c3-91e4-97d8e432f889/volumes" Nov 03 04:02:51.405674 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.405621 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" podUID="362725bd-ff9c-4608-9fc5-fe715d6a859d" containerName="main" containerID="cri-o://dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3" gracePeriod=45 Nov 03 04:02:51.693498 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.693469 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 04:02:51.789591 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789559 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-model-cache\") pod \"362725bd-ff9c-4608-9fc5-fe715d6a859d\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " Nov 03 04:02:51.789786 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789647 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-home\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-home\") pod \"362725bd-ff9c-4608-9fc5-fe715d6a859d\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " Nov 03 04:02:51.789786 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789689 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-tmp-dir\") pod \"362725bd-ff9c-4608-9fc5-fe715d6a859d\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " Nov 03 04:02:51.789786 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789709 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-dshm\") pod \"362725bd-ff9c-4608-9fc5-fe715d6a859d\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " Nov 03 04:02:51.789786 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789742 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362725bd-ff9c-4608-9fc5-fe715d6a859d-kube-api-access-ckbn2\" (UniqueName: \"kubernetes.io/projected/362725bd-ff9c-4608-9fc5-fe715d6a859d-kube-api-access-ckbn2\") pod \"362725bd-ff9c-4608-9fc5-fe715d6a859d\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " Nov 03 04:02:51.789786 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789780 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/362725bd-ff9c-4608-9fc5-fe715d6a859d-tls-certs\" (UniqueName: \"kubernetes.io/secret/362725bd-ff9c-4608-9fc5-fe715d6a859d-tls-certs\") pod \"362725bd-ff9c-4608-9fc5-fe715d6a859d\" (UID: \"362725bd-ff9c-4608-9fc5-fe715d6a859d\") " Nov 03 04:02:51.790074 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789906 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-home" pod "362725bd-ff9c-4608-9fc5-fe715d6a859d" (UID: "362725bd-ff9c-4608-9fc5-fe715d6a859d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:51.790074 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789895 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-model-cache" pod "362725bd-ff9c-4608-9fc5-fe715d6a859d" (UID: "362725bd-ff9c-4608-9fc5-fe715d6a859d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:51.790074 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.789949 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-tmp-dir" pod "362725bd-ff9c-4608-9fc5-fe715d6a859d" (UID: "362725bd-ff9c-4608-9fc5-fe715d6a859d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:51.790228 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.790209 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:51.790278 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.790229 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:51.790278 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.790244 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:51.792482 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.792435 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362725bd-ff9c-4608-9fc5-fe715d6a859d-tls-certs" pod "362725bd-ff9c-4608-9fc5-fe715d6a859d" (UID: "362725bd-ff9c-4608-9fc5-fe715d6a859d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:51.792626 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.792495 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-dshm" pod "362725bd-ff9c-4608-9fc5-fe715d6a859d" (UID: "362725bd-ff9c-4608-9fc5-fe715d6a859d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:51.792626 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.792519 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362725bd-ff9c-4608-9fc5-fe715d6a859d-kube-api-access-ckbn2" pod "362725bd-ff9c-4608-9fc5-fe715d6a859d" (UID: "362725bd-ff9c-4608-9fc5-fe715d6a859d"). InnerVolumeSpecName "kube-api-access-ckbn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:51.792997 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.792975 5117 generic.go:356] "Generic (PLEG): container finished" podID="362725bd-ff9c-4608-9fc5-fe715d6a859d" containerID="dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3" exitCode=0 Nov 03 04:02:51.793090 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.793035 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" Nov 03 04:02:51.793090 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.793051 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" event={"ID":"362725bd-ff9c-4608-9fc5-fe715d6a859d","Type":"ContainerDied","Data":"dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3"} Nov 03 04:02:51.793181 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.793099 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz" event={"ID":"362725bd-ff9c-4608-9fc5-fe715d6a859d","Type":"ContainerDied","Data":"60e11240762b8b92e944da7c6a18107d7eb6689cd5235057e436976ef55de158"} Nov 03 04:02:51.793181 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.793119 5117 scope.go:122] "RemoveContainer" containerID="dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3" Nov 03 04:02:51.825330 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.825304 5117 scope.go:122] "RemoveContainer" containerID="dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3" Nov 03 04:02:51.825750 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:02:51.825721 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3\": container with ID starting with dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3 not found: ID does not exist" containerID="dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3" Nov 03 04:02:51.825823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.825766 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3"} err="failed to get container status \"dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3\": rpc error: code = NotFound desc = could not find container \"dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3\": container with ID starting with dfd65a3685b2f77941f325f5edc5a92803c14cf2d6718c1c7563b0720e1bf9e3 not found: ID does not exist" Nov 03 04:02:51.891857 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.891825 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/362725bd-ff9c-4608-9fc5-fe715d6a859d-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:51.891857 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.891851 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckbn2\" (UniqueName: \"kubernetes.io/projected/362725bd-ff9c-4608-9fc5-fe715d6a859d-kube-api-access-ckbn2\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:51.891857 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.891864 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362725bd-ff9c-4608-9fc5-fe715d6a859d-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:51.983895 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.983774 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz"] Nov 03 04:02:51.987218 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:51.987191 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-10120d/auth-test-model-10120d-kserve-6f75969b69-b9zzz"] Nov 03 04:02:52.350141 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.350096 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" podUID="c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" containerName="main" containerID="cri-o://86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad" gracePeriod=45 Nov 03 04:02:52.618325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.618266 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 04:02:52.701484 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701419 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-dshm\") pod \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " Nov 03 04:02:52.701661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701493 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tls-certs\") pod \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " Nov 03 04:02:52.701661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701558 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tmp-dir\") pod \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " Nov 03 04:02:52.701661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701591 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-home\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-home\") pod \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " Nov 03 04:02:52.701661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701647 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-kube-api-access-zdng8\" (UniqueName: \"kubernetes.io/projected/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-kube-api-access-zdng8\") pod \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " Nov 03 04:02:52.701846 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701752 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-model-cache\") pod \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\" (UID: \"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a\") " Nov 03 04:02:52.701909 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701838 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tmp-dir" pod "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" (UID: "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:52.701909 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.701869 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-home" pod "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" (UID: "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:52.702083 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.702054 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-model-cache" pod "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" (UID: "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:52.702416 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.702388 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:52.702416 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.702412 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:52.702551 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.702428 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:52.703972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.703947 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-dshm" pod "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" (UID: "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:52.704098 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.703982 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-kube-api-access-zdng8" pod "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" (UID: "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a"). InnerVolumeSpecName "kube-api-access-zdng8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:52.704098 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.704023 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tls-certs" pod "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" (UID: "c8f48b59-3ae9-40bd-b29b-f420b50f6c2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:52.800531 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.800496 5117 generic.go:356] "Generic (PLEG): container finished" podID="c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" containerID="86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad" exitCode=0 Nov 03 04:02:52.800720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.800556 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" Nov 03 04:02:52.800720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.800577 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" event={"ID":"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a","Type":"ContainerDied","Data":"86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad"} Nov 03 04:02:52.800720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.800629 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b" event={"ID":"c8f48b59-3ae9-40bd-b29b-f420b50f6c2a","Type":"ContainerDied","Data":"2d566bc7cdd178b50f8599091a07dcaaa450ade1c992412d43a3a36bf1b19374"} Nov 03 04:02:52.800720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.800651 5117 scope.go:122] "RemoveContainer" containerID="86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad" Nov 03 04:02:52.803313 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.803291 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:52.803434 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.803317 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:52.803434 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.803331 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdng8\" (UniqueName: \"kubernetes.io/projected/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a-kube-api-access-zdng8\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:52.812630 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.812611 5117 scope.go:122] "RemoveContainer" containerID="86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad" Nov 03 04:02:52.812863 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:02:52.812844 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad\": container with ID starting with 86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad not found: ID does not exist" containerID="86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad" Nov 03 04:02:52.812922 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.812873 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad"} err="failed to get container status \"86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad\": rpc error: code = NotFound desc = could not find container \"86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad\": container with ID starting with 86bf9aa40689b632031e0284fdb4234dce5bab4ab7ae7fad18bb490d6268bcad not found: ID does not exist" Nov 03 04:02:52.968263 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.968177 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b"] Nov 03 04:02:52.974192 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:52.974165 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f3e02b/auth-test-model-f3e02b-kserve-698f6c9f57-gjk4b"] Nov 03 04:02:53.091211 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.091176 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362725bd-ff9c-4608-9fc5-fe715d6a859d" path="/var/lib/kubelet/pods/362725bd-ff9c-4608-9fc5-fe715d6a859d/volumes" Nov 03 04:02:53.091747 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.091723 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" path="/var/lib/kubelet/pods/c8f48b59-3ae9-40bd-b29b-f420b50f6c2a/volumes" Nov 03 04:02:53.336724 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.336680 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" podUID="648b8c29-69fd-4406-9b00-84ce1e72577e" containerName="main" containerID="cri-o://7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372" gracePeriod=45 Nov 03 04:02:53.615621 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.615595 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 04:02:53.713725 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.713691 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-home\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-home\") pod \"648b8c29-69fd-4406-9b00-84ce1e72577e\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.713804 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648b8c29-69fd-4406-9b00-84ce1e72577e-kube-api-access-gls49\" (UniqueName: \"kubernetes.io/projected/648b8c29-69fd-4406-9b00-84ce1e72577e-kube-api-access-gls49\") pod \"648b8c29-69fd-4406-9b00-84ce1e72577e\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.713830 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-tmp-dir\") pod \"648b8c29-69fd-4406-9b00-84ce1e72577e\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.713859 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-model-cache\") pod \"648b8c29-69fd-4406-9b00-84ce1e72577e\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.713894 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648b8c29-69fd-4406-9b00-84ce1e72577e-tls-certs\" (UniqueName: \"kubernetes.io/secret/648b8c29-69fd-4406-9b00-84ce1e72577e-tls-certs\") pod \"648b8c29-69fd-4406-9b00-84ce1e72577e\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.713916 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-dshm\") pod \"648b8c29-69fd-4406-9b00-84ce1e72577e\" (UID: \"648b8c29-69fd-4406-9b00-84ce1e72577e\") " Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.714030 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-home" pod "648b8c29-69fd-4406-9b00-84ce1e72577e" (UID: "648b8c29-69fd-4406-9b00-84ce1e72577e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.714115 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-model-cache" pod "648b8c29-69fd-4406-9b00-84ce1e72577e" (UID: "648b8c29-69fd-4406-9b00-84ce1e72577e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:53.714193 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.714130 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-tmp-dir" pod "648b8c29-69fd-4406-9b00-84ce1e72577e" (UID: "648b8c29-69fd-4406-9b00-84ce1e72577e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:53.714568 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.714349 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:53.714568 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.714364 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:53.714568 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.714375 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:53.716289 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.716265 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648b8c29-69fd-4406-9b00-84ce1e72577e-kube-api-access-gls49" pod "648b8c29-69fd-4406-9b00-84ce1e72577e" (UID: "648b8c29-69fd-4406-9b00-84ce1e72577e"). InnerVolumeSpecName "kube-api-access-gls49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:02:53.716405 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.716305 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-dshm" pod "648b8c29-69fd-4406-9b00-84ce1e72577e" (UID: "648b8c29-69fd-4406-9b00-84ce1e72577e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:02:53.716405 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.716335 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648b8c29-69fd-4406-9b00-84ce1e72577e-tls-certs" pod "648b8c29-69fd-4406-9b00-84ce1e72577e" (UID: "648b8c29-69fd-4406-9b00-84ce1e72577e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:02:53.809921 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.809879 5117 generic.go:356] "Generic (PLEG): container finished" podID="648b8c29-69fd-4406-9b00-84ce1e72577e" containerID="7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372" exitCode=0 Nov 03 04:02:53.810092 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.809937 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" event={"ID":"648b8c29-69fd-4406-9b00-84ce1e72577e","Type":"ContainerDied","Data":"7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372"} Nov 03 04:02:53.810092 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.809947 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" Nov 03 04:02:53.810092 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.809977 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d" event={"ID":"648b8c29-69fd-4406-9b00-84ce1e72577e","Type":"ContainerDied","Data":"f222ad91587cd3904c9ed5b68bad8af16da93e4a83b542bc085b7a6006e839a7"} Nov 03 04:02:53.810092 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.809997 5117 scope.go:122] "RemoveContainer" containerID="7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372" Nov 03 04:02:53.815160 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.815136 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gls49\" (UniqueName: \"kubernetes.io/projected/648b8c29-69fd-4406-9b00-84ce1e72577e-kube-api-access-gls49\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:53.815286 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.815165 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/648b8c29-69fd-4406-9b00-84ce1e72577e-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:53.815286 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.815175 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/648b8c29-69fd-4406-9b00-84ce1e72577e-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:02:53.822957 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.822878 5117 scope.go:122] "RemoveContainer" containerID="7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372" Nov 03 04:02:53.823479 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:02:53.823422 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372\": container with ID starting with 7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372 not found: ID does not exist" containerID="7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372" Nov 03 04:02:53.823586 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.823484 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372"} err="failed to get container status \"7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372\": rpc error: code = NotFound desc = could not find container \"7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372\": container with ID starting with 7244c2e289a3c4f77a0aa7b0e59ad63eb376a2496495d406592489e1154e7372 not found: ID does not exist" Nov 03 04:02:53.980521 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.980408 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d"] Nov 03 04:02:53.984589 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:53.984559 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/auth-test-model-66d902-kserve-84ccc54dd9-sw42d"] Nov 03 04:02:55.091699 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:02:55.091664 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648b8c29-69fd-4406-9b00-84ce1e72577e" path="/var/lib/kubelet/pods/648b8c29-69fd-4406-9b00-84ce1e72577e/volumes" Nov 03 04:03:00.011212 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:00.011163 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:03:09.111812 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.111776 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7119-a2d66e"] Nov 03 04:03:09.112347 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112331 5117 cpu_manager.go:414] "containerMap: removing container" podUID="c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" containerName="main" Nov 03 04:03:09.112347 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112349 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" containerName="main" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112368 5117 cpu_manager.go:414] "containerMap: removing container" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112377 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112407 5117 cpu_manager.go:414] "containerMap: removing container" podUID="362725bd-ff9c-4608-9fc5-fe715d6a859d" containerName="main" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112413 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362725bd-ff9c-4608-9fc5-fe715d6a859d" containerName="main" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112428 5117 cpu_manager.go:414] "containerMap: removing container" podUID="70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" containerName="maas-api" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112433 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" containerName="maas-api" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112460 5117 cpu_manager.go:414] "containerMap: removing container" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112467 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" Nov 03 04:03:09.112473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112474 5117 cpu_manager.go:414] "containerMap: removing container" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112479 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112489 5117 cpu_manager.go:414] "containerMap: removing container" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112493 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112501 5117 cpu_manager.go:414] "containerMap: removing container" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112505 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112514 5117 cpu_manager.go:414] "containerMap: removing container" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112518 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112536 5117 cpu_manager.go:414] "containerMap: removing container" podUID="648b8c29-69fd-4406-9b00-84ce1e72577e" containerName="main" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112542 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648b8c29-69fd-4406-9b00-84ce1e72577e" containerName="main" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112642 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f48b59-3ae9-40bd-b29b-f420b50f6c2a" containerName="main" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112651 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cad53d-ab1f-47c3-91e4-97d8e432f889" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112657 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="362725bd-ff9c-4608-9fc5-fe715d6a859d" containerName="main" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112665 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d4b3dd-8f85-4dd9-a4c0-46f0b0a3937e" containerName="maas-api" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112670 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b5de7a-3fb2-4bb3-8e5a-eacbb7f6897b" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112676 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="604d0d93-538e-498a-a3dc-b819b663eb30" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112682 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="3660a45d-bddf-4804-a5af-d6c9050bfe1f" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112687 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="15074a77-6cf3-4d5b-b9f5-e275361bee20" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112693 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="96310c43-9954-4cfe-907a-df1272d49305" containerName="istio-proxy" Nov 03 04:03:09.112749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.112699 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="648b8c29-69fd-4406-9b00-84ce1e72577e" containerName="main" Nov 03 04:03:09.117337 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.117318 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:09.174059 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.174026 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-a2d66e"] Nov 03 04:03:09.194976 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.194932 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5v9\" (UniqueName: \"kubernetes.io/projected/04874d5d-acc3-4abc-8e44-3a2796e68c4f-kube-api-access-wc5v9\") pod \"test-curl-7119-a2d66e\" (UID: \"04874d5d-acc3-4abc-8e44-3a2796e68c4f\") " pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:09.220893 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.220841 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr9t9\"" Nov 03 04:03:09.296045 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.296008 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5v9\" (UniqueName: \"kubernetes.io/projected/04874d5d-acc3-4abc-8e44-3a2796e68c4f-kube-api-access-wc5v9\") pod \"test-curl-7119-a2d66e\" (UID: \"04874d5d-acc3-4abc-8e44-3a2796e68c4f\") " pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:09.359265 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.359232 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5v9\" (UniqueName: \"kubernetes.io/projected/04874d5d-acc3-4abc-8e44-3a2796e68c4f-kube-api-access-wc5v9\") pod \"test-curl-7119-a2d66e\" (UID: \"04874d5d-acc3-4abc-8e44-3a2796e68c4f\") " pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:09.430540 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.430399 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:09.812500 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.812472 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-a2d66e"] Nov 03 04:03:09.813397 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:03:09.813370 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04874d5d_acc3_4abc_8e44_3a2796e68c4f.slice/crio-2c1a70547caf63ca1683cc7eb519361cfdb5645699d50cbb301e4903ae122869 WatchSource:0}: Error finding container 2c1a70547caf63ca1683cc7eb519361cfdb5645699d50cbb301e4903ae122869: Status 404 returned error can't find the container with id 2c1a70547caf63ca1683cc7eb519361cfdb5645699d50cbb301e4903ae122869 Nov 03 04:03:09.909229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.909187 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-a2d66e" event={"ID":"04874d5d-acc3-4abc-8e44-3a2796e68c4f","Type":"ContainerStarted","Data":"04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280"} Nov 03 04:03:09.909388 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:09.909239 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-a2d66e" event={"ID":"04874d5d-acc3-4abc-8e44-3a2796e68c4f","Type":"ContainerStarted","Data":"2c1a70547caf63ca1683cc7eb519361cfdb5645699d50cbb301e4903ae122869"} Nov 03 04:03:10.031018 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:10.030963 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7119-a2d66e" podStartSLOduration=1.030947755 podStartE2EDuration="1.030947755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:10.029843601 +0000 UTC m=+3725.060442140" watchObservedRunningTime="2026-11-03 04:03:10.030947755 +0000 UTC m=+3725.061546292" Nov 03 04:03:11.528218 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.528179 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7119-a2d66e"] Nov 03 04:03:11.532326 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.532299 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7119-a2d66e"] Nov 03 04:03:11.792510 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.792423 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7119-f896e3"] Nov 03 04:03:11.792965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.792950 5117 cpu_manager.go:414] "containerMap: removing container" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" containerName="test-curl-7119-a2d66e" Nov 03 04:03:11.793011 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.792967 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" containerName="test-curl-7119-a2d66e" Nov 03 04:03:11.793090 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.793080 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" containerName="test-curl-7119-a2d66e" Nov 03 04:03:11.796539 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.796522 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:11.821161 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.821128 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fx9\" (UniqueName: \"kubernetes.io/projected/5aca10e2-c6b6-4429-a817-b51724935ce7-kube-api-access-47fx9\") pod \"test-curl-7119-f896e3\" (UID: \"5aca10e2-c6b6-4429-a817-b51724935ce7\") " pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:11.852143 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.852115 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-f896e3"] Nov 03 04:03:11.920958 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.920909 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7119-a2d66e" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" containerName="test-curl-7119-a2d66e" containerID="cri-o://04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280" gracePeriod=2 Nov 03 04:03:11.922720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.922691 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fx9\" (UniqueName: \"kubernetes.io/projected/5aca10e2-c6b6-4429-a817-b51724935ce7-kube-api-access-47fx9\") pod \"test-curl-7119-f896e3\" (UID: \"5aca10e2-c6b6-4429-a817-b51724935ce7\") " pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:11.973298 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:11.973258 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-a2d66e\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:11.984484 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:11.984432 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fx9\" (UniqueName: \"kubernetes.io/projected/5aca10e2-c6b6-4429-a817-b51724935ce7-kube-api-access-47fx9\") pod \"test-curl-7119-f896e3\" (UID: \"5aca10e2-c6b6-4429-a817-b51724935ce7\") " pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:12.108403 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:12.108369 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:12.481468 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:12.481425 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-f896e3"] Nov 03 04:03:12.482846 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:03:12.482816 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aca10e2_c6b6_4429_a817_b51724935ce7.slice/crio-de67677eb22c8814d47ceb039c025bbc74855c4e331c795fdb36a784a1375383 WatchSource:0}: Error finding container de67677eb22c8814d47ceb039c025bbc74855c4e331c795fdb36a784a1375383: Status 404 returned error can't find the container with id de67677eb22c8814d47ceb039c025bbc74855c4e331c795fdb36a784a1375383 Nov 03 04:03:12.928458 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:12.928405 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-f896e3" event={"ID":"5aca10e2-c6b6-4429-a817-b51724935ce7","Type":"ContainerStarted","Data":"70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b"} Nov 03 04:03:12.928458 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:12.928459 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-f896e3" event={"ID":"5aca10e2-c6b6-4429-a817-b51724935ce7","Type":"ContainerStarted","Data":"de67677eb22c8814d47ceb039c025bbc74855c4e331c795fdb36a784a1375383"} Nov 03 04:03:13.046149 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:13.046085 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7119-f896e3" podStartSLOduration=2.0460667790000002 podStartE2EDuration="2.046066779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:13.044148008 +0000 UTC m=+3728.074746545" watchObservedRunningTime="2026-11-03 04:03:13.046066779 +0000 UTC m=+3728.076665319" Nov 03 04:03:13.096286 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:13.096247 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-a2d66e\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:14.159120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.159091 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:14.246482 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.246349 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04874d5d-acc3-4abc-8e44-3a2796e68c4f-kube-api-access-wc5v9\" (UniqueName: \"kubernetes.io/projected/04874d5d-acc3-4abc-8e44-3a2796e68c4f-kube-api-access-wc5v9\") pod \"04874d5d-acc3-4abc-8e44-3a2796e68c4f\" (UID: \"04874d5d-acc3-4abc-8e44-3a2796e68c4f\") " Nov 03 04:03:14.248729 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.248701 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04874d5d-acc3-4abc-8e44-3a2796e68c4f-kube-api-access-wc5v9" pod "04874d5d-acc3-4abc-8e44-3a2796e68c4f" (UID: "04874d5d-acc3-4abc-8e44-3a2796e68c4f"). InnerVolumeSpecName "kube-api-access-wc5v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:03:14.347531 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.347493 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5v9\" (UniqueName: \"kubernetes.io/projected/04874d5d-acc3-4abc-8e44-3a2796e68c4f-kube-api-access-wc5v9\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:14.625799 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.625764 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7119-f896e3"] Nov 03 04:03:14.629851 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.629819 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7119-f896e3"] Nov 03 04:03:14.790600 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.790549 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7119-25d85f"] Nov 03 04:03:14.791890 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.791854 5117 cpu_manager.go:414] "containerMap: removing container" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" containerName="test-curl-7119-f896e3" Nov 03 04:03:14.791890 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.791889 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" containerName="test-curl-7119-f896e3" Nov 03 04:03:14.792185 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.792168 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" containerName="test-curl-7119-f896e3" Nov 03 04:03:14.796784 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.796759 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:14.846436 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.846402 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-25d85f"] Nov 03 04:03:14.854616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.854578 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2tl\" (UniqueName: \"kubernetes.io/projected/7c5568f6-6aea-4209-a23c-fce21da3ee2f-kube-api-access-zm2tl\") pod \"test-curl-7119-25d85f\" (UID: \"7c5568f6-6aea-4209-a23c-fce21da3ee2f\") " pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:14.940366 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.940273 5117 generic.go:356] "Generic (PLEG): container finished" podID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" containerID="04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280" exitCode=137 Nov 03 04:03:14.940366 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.940354 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:14.940596 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.940382 5117 scope.go:122] "RemoveContainer" containerID="04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280" Nov 03 04:03:14.940596 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.940427 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7119-f896e3" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" containerName="test-curl-7119-f896e3" containerID="cri-o://70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b" gracePeriod=2 Nov 03 04:03:14.950784 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.950755 5117 scope.go:122] "RemoveContainer" containerID="04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280" Nov 03 04:03:14.951103 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:14.951083 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280\": container with ID starting with 04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280 not found: ID does not exist" containerID="04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280" Nov 03 04:03:14.951152 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.951114 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280"} err="failed to get container status \"04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280\": rpc error: code = NotFound desc = could not find container \"04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280\": container with ID starting with 04451e13841d2c6beec609edb93cad48ba525a33d8291a5a71b4459225fc0280 not found: ID does not exist" Nov 03 04:03:14.955745 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:14.955722 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2tl\" (UniqueName: \"kubernetes.io/projected/7c5568f6-6aea-4209-a23c-fce21da3ee2f-kube-api-access-zm2tl\") pod \"test-curl-7119-25d85f\" (UID: \"7c5568f6-6aea-4209-a23c-fce21da3ee2f\") " pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:14.993035 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:14.992993 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-f896e3\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:15.017954 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:15.017916 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2tl\" (UniqueName: \"kubernetes.io/projected/7c5568f6-6aea-4209-a23c-fce21da3ee2f-kube-api-access-zm2tl\") pod \"test-curl-7119-25d85f\" (UID: \"7c5568f6-6aea-4209-a23c-fce21da3ee2f\") " pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:15.045470 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:15.045400 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-f896e3\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:15.090997 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:15.090959 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" path="/var/lib/kubelet/pods/04874d5d-acc3-4abc-8e44-3a2796e68c4f/volumes" Nov 03 04:03:15.097613 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:15.097579 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-a2d66e\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="04874d5d-acc3-4abc-8e44-3a2796e68c4f" pod="opendatahub/test-curl-7119-a2d66e" Nov 03 04:03:15.109821 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:15.109791 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:15.489292 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:15.489264 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-25d85f"] Nov 03 04:03:15.491535 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:03:15.491503 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c5568f6_6aea_4209_a23c_fce21da3ee2f.slice/crio-32680a41f3270b0738641e1bcd2fa17c9fb14191c5238d011bba96ef148d700a WatchSource:0}: Error finding container 32680a41f3270b0738641e1bcd2fa17c9fb14191c5238d011bba96ef148d700a: Status 404 returned error can't find the container with id 32680a41f3270b0738641e1bcd2fa17c9fb14191c5238d011bba96ef148d700a Nov 03 04:03:15.948157 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:15.948115 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-25d85f" event={"ID":"7c5568f6-6aea-4209-a23c-fce21da3ee2f","Type":"ContainerStarted","Data":"2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3"} Nov 03 04:03:15.948157 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:15.948173 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-25d85f" event={"ID":"7c5568f6-6aea-4209-a23c-fce21da3ee2f","Type":"ContainerStarted","Data":"32680a41f3270b0738641e1bcd2fa17c9fb14191c5238d011bba96ef148d700a"} Nov 03 04:03:16.064857 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:16.064797 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7119-25d85f" podStartSLOduration=2.064781032 podStartE2EDuration="2.064781032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:16.063686337 +0000 UTC m=+3731.094284876" watchObservedRunningTime="2026-11-03 04:03:16.064781032 +0000 UTC m=+3731.095379570" Nov 03 04:03:17.174779 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.174754 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:17.227022 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:17.226941 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-f896e3\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:17.279608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.279576 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aca10e2-c6b6-4429-a817-b51724935ce7-kube-api-access-47fx9\" (UniqueName: \"kubernetes.io/projected/5aca10e2-c6b6-4429-a817-b51724935ce7-kube-api-access-47fx9\") pod \"5aca10e2-c6b6-4429-a817-b51724935ce7\" (UID: \"5aca10e2-c6b6-4429-a817-b51724935ce7\") " Nov 03 04:03:17.281705 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.281671 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aca10e2-c6b6-4429-a817-b51724935ce7-kube-api-access-47fx9" pod "5aca10e2-c6b6-4429-a817-b51724935ce7" (UID: "5aca10e2-c6b6-4429-a817-b51724935ce7"). InnerVolumeSpecName "kube-api-access-47fx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:03:17.381361 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.381330 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fx9\" (UniqueName: \"kubernetes.io/projected/5aca10e2-c6b6-4429-a817-b51724935ce7-kube-api-access-47fx9\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:17.495987 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.495907 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7119-25d85f"] Nov 03 04:03:17.500560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.500533 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7119-25d85f"] Nov 03 04:03:17.657263 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.657228 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7119-1578e4"] Nov 03 04:03:17.657813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.657790 5117 cpu_manager.go:414] "containerMap: removing container" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" containerName="test-curl-7119-25d85f" Nov 03 04:03:17.657813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.657809 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" containerName="test-curl-7119-25d85f" Nov 03 04:03:17.657975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.657928 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" containerName="test-curl-7119-25d85f" Nov 03 04:03:17.661196 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.661180 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:17.684333 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.684300 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgwq\" (UniqueName: \"kubernetes.io/projected/bb0b1c7f-667f-4022-a6fc-e79c66a3decf-kube-api-access-ncgwq\") pod \"test-curl-7119-1578e4\" (UID: \"bb0b1c7f-667f-4022-a6fc-e79c66a3decf\") " pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:17.716903 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.716871 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-1578e4"] Nov 03 04:03:17.785174 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.785083 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgwq\" (UniqueName: \"kubernetes.io/projected/bb0b1c7f-667f-4022-a6fc-e79c66a3decf-kube-api-access-ncgwq\") pod \"test-curl-7119-1578e4\" (UID: \"bb0b1c7f-667f-4022-a6fc-e79c66a3decf\") " pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:17.827921 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:17.827886 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-f896e3\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:17.847489 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.847439 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgwq\" (UniqueName: \"kubernetes.io/projected/bb0b1c7f-667f-4022-a6fc-e79c66a3decf-kube-api-access-ncgwq\") pod \"test-curl-7119-1578e4\" (UID: \"bb0b1c7f-667f-4022-a6fc-e79c66a3decf\") " pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:17.963480 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.963424 5117 generic.go:356] "Generic (PLEG): container finished" podID="5aca10e2-c6b6-4429-a817-b51724935ce7" containerID="70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b" exitCode=137 Nov 03 04:03:17.963678 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.963489 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:17.963678 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.963534 5117 scope.go:122] "RemoveContainer" containerID="70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b" Nov 03 04:03:17.963678 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.963664 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7119-25d85f" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" containerName="test-curl-7119-25d85f" containerID="cri-o://2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3" gracePeriod=2 Nov 03 04:03:17.973015 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.972988 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:17.973471 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.973279 5117 scope.go:122] "RemoveContainer" containerID="70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b" Nov 03 04:03:17.973805 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:17.973777 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b\": container with ID starting with 70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b not found: ID does not exist" containerID="70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b" Nov 03 04:03:17.973873 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:17.973818 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b"} err="failed to get container status \"70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b\": rpc error: code = NotFound desc = could not find container \"70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b\": container with ID starting with 70d8c114a5627cc090988fc238674b1fe703cc0ba935a02dc4a809c814368d8b not found: ID does not exist" Nov 03 04:03:18.016109 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:18.016064 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-25d85f\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:18.068541 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:18.068493 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-f896e3\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:18.120951 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:18.120911 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-f896e3\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:18.173226 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:18.173186 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-25d85f\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:18.351578 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:18.351545 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7119-1578e4"] Nov 03 04:03:18.351955 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:03:18.351786 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb0b1c7f_667f_4022_a6fc_e79c66a3decf.slice/crio-eb9dde1f494ce47f762a0b93067f633896f6fad2e87ad2a9c859e58d1517ecb1 WatchSource:0}: Error finding container eb9dde1f494ce47f762a0b93067f633896f6fad2e87ad2a9c859e58d1517ecb1: Status 404 returned error can't find the container with id eb9dde1f494ce47f762a0b93067f633896f6fad2e87ad2a9c859e58d1517ecb1 Nov 03 04:03:18.971375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:18.971327 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-1578e4" event={"ID":"bb0b1c7f-667f-4022-a6fc-e79c66a3decf","Type":"ContainerStarted","Data":"d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71"} Nov 03 04:03:18.971375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:18.971378 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7119-1578e4" event={"ID":"bb0b1c7f-667f-4022-a6fc-e79c66a3decf","Type":"ContainerStarted","Data":"eb9dde1f494ce47f762a0b93067f633896f6fad2e87ad2a9c859e58d1517ecb1"} Nov 03 04:03:19.089205 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:19.089154 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7119-1578e4" podStartSLOduration=2.089140465 podStartE2EDuration="2.089140465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:19.086869621 +0000 UTC m=+3734.117468159" watchObservedRunningTime="2026-11-03 04:03:19.089140465 +0000 UTC m=+3734.119739003" Nov 03 04:03:19.090958 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:19.090931 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" path="/var/lib/kubelet/pods/5aca10e2-c6b6-4429-a817-b51724935ce7/volumes" Nov 03 04:03:19.139160 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:19.139121 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-f896e3\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="5aca10e2-c6b6-4429-a817-b51724935ce7" pod="opendatahub/test-curl-7119-f896e3" Nov 03 04:03:19.191547 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:19.191507 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-25d85f\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:20.201192 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.201165 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:20.312801 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.312769 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5568f6-6aea-4209-a23c-fce21da3ee2f-kube-api-access-zm2tl\" (UniqueName: \"kubernetes.io/projected/7c5568f6-6aea-4209-a23c-fce21da3ee2f-kube-api-access-zm2tl\") pod \"7c5568f6-6aea-4209-a23c-fce21da3ee2f\" (UID: \"7c5568f6-6aea-4209-a23c-fce21da3ee2f\") " Nov 03 04:03:20.315016 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.314993 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5568f6-6aea-4209-a23c-fce21da3ee2f-kube-api-access-zm2tl" pod "7c5568f6-6aea-4209-a23c-fce21da3ee2f" (UID: "7c5568f6-6aea-4209-a23c-fce21da3ee2f"). InnerVolumeSpecName "kube-api-access-zm2tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:03:20.414378 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.414334 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2tl\" (UniqueName: \"kubernetes.io/projected/7c5568f6-6aea-4209-a23c-fce21da3ee2f-kube-api-access-zm2tl\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:20.631389 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.631300 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7119-1578e4"] Nov 03 04:03:20.638303 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.638265 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7119-1578e4"] Nov 03 04:03:20.987816 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.987717 5117 generic.go:356] "Generic (PLEG): container finished" podID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" containerID="2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3" exitCode=137 Nov 03 04:03:20.987816 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.987770 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:20.988057 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.987829 5117 scope.go:122] "RemoveContainer" containerID="2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3" Nov 03 04:03:20.988148 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:20.988118 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7119-1578e4" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" containerName="test-curl-7119-1578e4" containerID="cri-o://d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71" gracePeriod=2 Nov 03 04:03:21.001145 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:21.001126 5117 scope.go:122] "RemoveContainer" containerID="2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3" Nov 03 04:03:21.001494 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:21.001472 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3\": container with ID starting with 2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3 not found: ID does not exist" containerID="2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3" Nov 03 04:03:21.001584 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:21.001509 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3"} err="failed to get container status \"2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3\": rpc error: code = NotFound desc = could not find container \"2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3\": container with ID starting with 2209563d3807cea05f1238a21ea11c269864f4d0d2a0f6225f94ee1748bb2be3 not found: ID does not exist" Nov 03 04:03:21.040379 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:21.040343 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-1578e4\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:21.090043 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:21.090005 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" path="/var/lib/kubelet/pods/7c5568f6-6aea-4209-a23c-fce21da3ee2f/volumes" Nov 03 04:03:21.092672 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:21.092637 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-25d85f\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="7c5568f6-6aea-4209-a23c-fce21da3ee2f" pod="opendatahub/test-curl-7119-25d85f" Nov 03 04:03:21.144999 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:21.144952 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-1578e4\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:23.244530 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:23.244474 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:23.346786 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:23.346755 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0b1c7f-667f-4022-a6fc-e79c66a3decf-kube-api-access-ncgwq\" (UniqueName: \"kubernetes.io/projected/bb0b1c7f-667f-4022-a6fc-e79c66a3decf-kube-api-access-ncgwq\") pod \"bb0b1c7f-667f-4022-a6fc-e79c66a3decf\" (UID: \"bb0b1c7f-667f-4022-a6fc-e79c66a3decf\") " Nov 03 04:03:23.349041 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:23.349006 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0b1c7f-667f-4022-a6fc-e79c66a3decf-kube-api-access-ncgwq" pod "bb0b1c7f-667f-4022-a6fc-e79c66a3decf" (UID: "bb0b1c7f-667f-4022-a6fc-e79c66a3decf"). InnerVolumeSpecName "kube-api-access-ncgwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:03:23.448207 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:23.448176 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgwq\" (UniqueName: \"kubernetes.io/projected/bb0b1c7f-667f-4022-a6fc-e79c66a3decf-kube-api-access-ncgwq\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:24.007791 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:24.007753 5117 generic.go:356] "Generic (PLEG): container finished" podID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" containerID="d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71" exitCode=137 Nov 03 04:03:24.008009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:24.007812 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:24.008009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:24.007863 5117 scope.go:122] "RemoveContainer" containerID="d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71" Nov 03 04:03:24.018594 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:24.018573 5117 scope.go:122] "RemoveContainer" containerID="d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71" Nov 03 04:03:24.021214 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:24.021178 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71\": container with ID starting with d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71 not found: ID does not exist" containerID="d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71" Nov 03 04:03:24.021311 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:24.021221 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71"} err="failed to get container status \"d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71\": rpc error: code = NotFound desc = could not find container \"d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71\": container with ID starting with d1a1765172c7748db75054d6996df83026342f815c8e4efc55d37eba4bf2ee71 not found: ID does not exist" Nov 03 04:03:24.080628 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:24.080581 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-1578e4\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:25.090034 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:25.089988 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" path="/var/lib/kubelet/pods/bb0b1c7f-667f-4022-a6fc-e79c66a3decf/volumes" Nov 03 04:03:25.137951 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:25.137891 5117 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7119-1578e4\" is forbidden: User \"system:node:ip-10-0-138-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-120.us-west-1.compute.internal' and this object" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" pod="opendatahub/test-curl-7119-1578e4" Nov 03 04:03:28.809701 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:28.809661 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z"] Nov 03 04:03:30.013684 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:30.013645 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:03:33.570150 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.570113 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8"] Nov 03 04:03:33.570765 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.570743 5117 cpu_manager.go:414] "containerMap: removing container" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" containerName="test-curl-7119-1578e4" Nov 03 04:03:33.570838 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.570767 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" containerName="test-curl-7119-1578e4" Nov 03 04:03:33.570947 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.570935 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0b1c7f-667f-4022-a6fc-e79c66a3decf" containerName="test-curl-7119-1578e4" Nov 03 04:03:33.574248 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.574232 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.656000 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.655930 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkk4\" (UniqueName: \"kubernetes.io/projected/714e799a-0d64-4598-aaa1-92ff5a63ff91-kube-api-access-qkkk4\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.656249 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.656099 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.656249 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.656139 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714e799a-0d64-4598-aaa1-92ff5a63ff91-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.656470 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.656314 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.680468 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.680406 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-03da9e-serving-cert\"" Nov 03 04:03:33.683150 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.683127 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8"] Nov 03 04:03:33.757791 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.757749 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.757959 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.757861 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkk4\" (UniqueName: \"kubernetes.io/projected/714e799a-0d64-4598-aaa1-92ff5a63ff91-kube-api-access-qkkk4\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.757959 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.757938 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.758099 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.757966 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714e799a-0d64-4598-aaa1-92ff5a63ff91-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.758661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.758639 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714e799a-0d64-4598-aaa1-92ff5a63ff91-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.759079 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:33.759056 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-serving-cert: secret "maas-api-e2e-shared-b-w3-03da9e-serving-cert" not found Nov 03 04:03:33.759209 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:33.759139 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls podName:714e799a-0d64-4598-aaa1-92ff5a63ff91 nodeName:}" failed. No retries permitted until 2026-11-03 04:03:34.259117915 +0000 UTC m=+3749.289716444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls") pod "maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" (UID: "714e799a-0d64-4598-aaa1-92ff5a63ff91") : secret "maas-api-e2e-shared-b-w3-03da9e-serving-cert" not found Nov 03 04:03:33.761393 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.761363 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:33.821428 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:33.821349 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkk4\" (UniqueName: \"kubernetes.io/projected/714e799a-0d64-4598-aaa1-92ff5a63ff91-kube-api-access-qkkk4\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:34.264049 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:34.263960 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:34.266481 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:34.266430 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:34.488128 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:34.488095 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:34.872617 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:34.869286 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8"] Nov 03 04:03:34.879758 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:34.879733 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 04:03:35.090694 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:35.090656 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" event={"ID":"714e799a-0d64-4598-aaa1-92ff5a63ff91","Type":"ContainerStarted","Data":"febdf900524c2c3549527fcf41065dff011156151e4782648fed3dcf53c10c80"} Nov 03 04:03:36.095475 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:36.095359 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" event={"ID":"714e799a-0d64-4598-aaa1-92ff5a63ff91","Type":"ContainerStarted","Data":"8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd"} Nov 03 04:03:36.095475 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:36.095424 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:36.217642 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:36.217583 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" podStartSLOduration=2.282630913 podStartE2EDuration="3.217568765s" totalImagesPullingTime="934.937852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:36.216513374 +0000 UTC m=+3751.247111913" watchObservedRunningTime="2026-11-03 04:03:36.217568765 +0000 UTC m=+3751.248167303" Nov 03 04:03:42.107934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:42.107901 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:03:43.833811 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:43.833756 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" podUID="1449b617-7602-4f4b-a0d4-4561158bd3df" containerName="main" containerID="cri-o://5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261" gracePeriod=45 Nov 03 04:03:44.099274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.099220 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 04:03:44.148566 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.148534 5117 generic.go:356] "Generic (PLEG): container finished" podID="1449b617-7602-4f4b-a0d4-4561158bd3df" containerID="5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261" exitCode=0 Nov 03 04:03:44.148733 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.148587 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" Nov 03 04:03:44.148733 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.148610 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" event={"ID":"1449b617-7602-4f4b-a0d4-4561158bd3df","Type":"ContainerDied","Data":"5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261"} Nov 03 04:03:44.148733 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.148653 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z" event={"ID":"1449b617-7602-4f4b-a0d4-4561158bd3df","Type":"ContainerDied","Data":"881c32b866dc182c8276b64c32bb0ffdf248ba043f3f17281a7d7db3f22a6b96"} Nov 03 04:03:44.148733 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.148670 5117 scope.go:122] "RemoveContainer" containerID="5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261" Nov 03 04:03:44.159280 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.159263 5117 scope.go:122] "RemoveContainer" containerID="5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261" Nov 03 04:03:44.159627 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:03:44.159597 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261\": container with ID starting with 5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261 not found: ID does not exist" containerID="5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261" Nov 03 04:03:44.159708 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.159644 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261"} err="failed to get container status \"5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261\": rpc error: code = NotFound desc = could not find container \"5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261\": container with ID starting with 5405c9466b9fb4b828b2acbccb1cacd77efd71fb45c880e0ef66c94ec3712261 not found: ID does not exist" Nov 03 04:03:44.275173 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275142 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1449b617-7602-4f4b-a0d4-4561158bd3df-tls-certs\" (UniqueName: \"kubernetes.io/secret/1449b617-7602-4f4b-a0d4-4561158bd3df-tls-certs\") pod \"1449b617-7602-4f4b-a0d4-4561158bd3df\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " Nov 03 04:03:44.275375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275286 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-tmp-dir\") pod \"1449b617-7602-4f4b-a0d4-4561158bd3df\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " Nov 03 04:03:44.275375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275324 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-dshm\") pod \"1449b617-7602-4f4b-a0d4-4561158bd3df\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " Nov 03 04:03:44.275375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275340 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-home\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-home\") pod \"1449b617-7602-4f4b-a0d4-4561158bd3df\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " Nov 03 04:03:44.275593 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275384 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-model-cache\") pod \"1449b617-7602-4f4b-a0d4-4561158bd3df\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " Nov 03 04:03:44.275593 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275427 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1449b617-7602-4f4b-a0d4-4561158bd3df-kube-api-access-75hgz\" (UniqueName: \"kubernetes.io/projected/1449b617-7602-4f4b-a0d4-4561158bd3df-kube-api-access-75hgz\") pod \"1449b617-7602-4f4b-a0d4-4561158bd3df\" (UID: \"1449b617-7602-4f4b-a0d4-4561158bd3df\") " Nov 03 04:03:44.275709 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275660 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-tmp-dir" pod "1449b617-7602-4f4b-a0d4-4561158bd3df" (UID: "1449b617-7602-4f4b-a0d4-4561158bd3df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:44.275709 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275676 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-home" pod "1449b617-7602-4f4b-a0d4-4561158bd3df" (UID: "1449b617-7602-4f4b-a0d4-4561158bd3df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:44.275839 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275726 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-model-cache" pod "1449b617-7602-4f4b-a0d4-4561158bd3df" (UID: "1449b617-7602-4f4b-a0d4-4561158bd3df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:44.275940 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275920 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:44.276004 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275943 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:44.276004 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.275957 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:44.277837 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.277811 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-dshm" pod "1449b617-7602-4f4b-a0d4-4561158bd3df" (UID: "1449b617-7602-4f4b-a0d4-4561158bd3df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:03:44.277962 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.277817 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1449b617-7602-4f4b-a0d4-4561158bd3df-tls-certs" pod "1449b617-7602-4f4b-a0d4-4561158bd3df" (UID: "1449b617-7602-4f4b-a0d4-4561158bd3df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:03:44.277962 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.277944 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1449b617-7602-4f4b-a0d4-4561158bd3df-kube-api-access-75hgz" pod "1449b617-7602-4f4b-a0d4-4561158bd3df" (UID: "1449b617-7602-4f4b-a0d4-4561158bd3df"). InnerVolumeSpecName "kube-api-access-75hgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:03:44.376937 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.376864 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449b617-7602-4f4b-a0d4-4561158bd3df-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:44.376937 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.376895 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75hgz\" (UniqueName: \"kubernetes.io/projected/1449b617-7602-4f4b-a0d4-4561158bd3df-kube-api-access-75hgz\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:44.376937 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.376906 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449b617-7602-4f4b-a0d4-4561158bd3df-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:03:44.572196 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.572161 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z"] Nov 03 04:03:44.576375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:44.576345 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-efa9bf/auth-test-model-efa9bf-kserve-6cbf89ffc7-js96z"] Nov 03 04:03:45.090476 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:03:45.090406 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1449b617-7602-4f4b-a0d4-4561158bd3df" path="/var/lib/kubelet/pods/1449b617-7602-4f4b-a0d4-4561158bd3df/volumes" Nov 03 04:04:00.010620 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:04:00.010582 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:04:08.889005 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:08.888962 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x"] Nov 03 04:04:08.889584 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:08.889570 5117 cpu_manager.go:414] "containerMap: removing container" podUID="1449b617-7602-4f4b-a0d4-4561158bd3df" containerName="main" Nov 03 04:04:08.889650 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:08.889587 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1449b617-7602-4f4b-a0d4-4561158bd3df" containerName="main" Nov 03 04:04:08.889711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:08.889699 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="1449b617-7602-4f4b-a0d4-4561158bd3df" containerName="main" Nov 03 04:04:08.894386 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:08.894368 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:08.999370 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:08.999339 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8c832d-serving-cert\"" Nov 03 04:04:09.002198 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.002171 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x"] Nov 03 04:04:09.020824 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.020794 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.020991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.020844 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8801da75-7d4d-463b-807b-5ba26b9e34fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.020991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.020967 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqsx8\" (UniqueName: \"kubernetes.io/projected/8801da75-7d4d-463b-807b-5ba26b9e34fa-kube-api-access-vqsx8\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.021153 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.021125 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.122510 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.122470 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.122815 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.122549 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.122815 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.122608 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8801da75-7d4d-463b-807b-5ba26b9e34fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.122815 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:04:09.122669 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-serving-cert: secret "maas-api-e2e-extmod-a-8c832d-serving-cert" not found Nov 03 04:04:09.122815 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.122700 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqsx8\" (UniqueName: \"kubernetes.io/projected/8801da75-7d4d-463b-807b-5ba26b9e34fa-kube-api-access-vqsx8\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.122815 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:04:09.122760 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls podName:8801da75-7d4d-463b-807b-5ba26b9e34fa nodeName:}" failed. No retries permitted until 2026-11-03 04:04:09.622734332 +0000 UTC m=+3784.653332865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls") pod "maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" (UID: "8801da75-7d4d-463b-807b-5ba26b9e34fa") : secret "maas-api-e2e-extmod-a-8c832d-serving-cert" not found Nov 03 04:04:09.123421 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.123398 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8801da75-7d4d-463b-807b-5ba26b9e34fa-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.125625 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.125600 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.189044 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.188959 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqsx8\" (UniqueName: \"kubernetes.io/projected/8801da75-7d4d-463b-807b-5ba26b9e34fa-kube-api-access-vqsx8\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.628034 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.627989 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.630755 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.630718 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:09.808120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:09.808081 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:10.193609 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:10.193578 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x"] Nov 03 04:04:10.195353 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:04:10.195323 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8801da75_7d4d_463b_807b_5ba26b9e34fa.slice/crio-b870c8ffc7cf1b2bc8647175d91081d1181e8c4407c59c1109632f9e16e56e3c WatchSource:0}: Error finding container b870c8ffc7cf1b2bc8647175d91081d1181e8c4407c59c1109632f9e16e56e3c: Status 404 returned error can't find the container with id b870c8ffc7cf1b2bc8647175d91081d1181e8c4407c59c1109632f9e16e56e3c Nov 03 04:04:10.304912 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:10.304878 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" event={"ID":"8801da75-7d4d-463b-807b-5ba26b9e34fa","Type":"ContainerStarted","Data":"b870c8ffc7cf1b2bc8647175d91081d1181e8c4407c59c1109632f9e16e56e3c"} Nov 03 04:04:11.317366 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:11.317329 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" event={"ID":"8801da75-7d4d-463b-807b-5ba26b9e34fa","Type":"ContainerStarted","Data":"ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9"} Nov 03 04:04:11.317807 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:11.317429 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:11.438242 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:11.438132 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" podStartSLOduration=2.559271296 podStartE2EDuration="3.438118522s" totalImagesPullingTime="878.847226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:04:11.437056179 +0000 UTC m=+3786.467654728" watchObservedRunningTime="2026-11-03 04:04:11.438118522 +0000 UTC m=+3786.468717060" Nov 03 04:04:17.329146 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:17.329118 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:04:18.784093 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.784052 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474"] Nov 03 04:04:18.789159 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.789138 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:18.843413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.843378 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a0afad-serving-cert\"" Nov 03 04:04:18.897038 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.897006 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474"] Nov 03 04:04:18.928472 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.928401 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:18.928615 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.928570 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:18.928676 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.928658 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpnh\" (UniqueName: \"kubernetes.io/projected/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-kube-api-access-wtpnh\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:18.928772 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:18.928756 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.030051 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.030005 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.030260 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.030103 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.030260 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.030171 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.030260 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.030229 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpnh\" (UniqueName: \"kubernetes.io/projected/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-kube-api-access-wtpnh\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.030637 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:04:19.030613 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-serving-cert: secret "maas-api-e2e-extmod-a-a0afad-serving-cert" not found Nov 03 04:04:19.030728 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:04:19.030687 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls podName:7f5e5bd9-7556-4b3d-91b0-5463c77d0016 nodeName:}" failed. No retries permitted until 2026-11-03 04:04:19.530665514 +0000 UTC m=+3794.561264043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls") pod "maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" (UID: "7f5e5bd9-7556-4b3d-91b0-5463c77d0016") : secret "maas-api-e2e-extmod-a-a0afad-serving-cert" not found Nov 03 04:04:19.031516 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.031488 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.033291 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.033270 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.094540 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.094504 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpnh\" (UniqueName: \"kubernetes.io/projected/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-kube-api-access-wtpnh\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.536125 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.536025 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.538567 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.538540 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:19.702544 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:19.702510 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:20.080720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:20.080694 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474"] Nov 03 04:04:20.082213 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:04:20.082180 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5e5bd9_7556_4b3d_91b0_5463c77d0016.slice/crio-b491522d56dc49dc9c864510ade00a7fbee8db141529b51331446b9a6cf66f45 WatchSource:0}: Error finding container b491522d56dc49dc9c864510ade00a7fbee8db141529b51331446b9a6cf66f45: Status 404 returned error can't find the container with id b491522d56dc49dc9c864510ade00a7fbee8db141529b51331446b9a6cf66f45 Nov 03 04:04:20.383845 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:20.383752 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" event={"ID":"7f5e5bd9-7556-4b3d-91b0-5463c77d0016","Type":"ContainerStarted","Data":"b491522d56dc49dc9c864510ade00a7fbee8db141529b51331446b9a6cf66f45"} Nov 03 04:04:21.393503 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:21.393467 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" event={"ID":"7f5e5bd9-7556-4b3d-91b0-5463c77d0016","Type":"ContainerStarted","Data":"02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803"} Nov 03 04:04:21.393948 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:21.393553 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:21.516363 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:21.516306 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" podStartSLOduration=2.7136216429999998 podStartE2EDuration="3.516289866s" totalImagesPullingTime="802.668223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:04:21.512875761 +0000 UTC m=+3796.543474299" watchObservedRunningTime="2026-11-03 04:04:21.516289866 +0000 UTC m=+3796.546888426" Nov 03 04:04:27.405341 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:27.405306 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:04:30.010623 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:04:30.010578 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:04:57.793860 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.793772 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp"] Nov 03 04:04:57.798238 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.798214 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:57.851592 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.851555 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-90139c-serving-cert\"" Nov 03 04:04:57.905368 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.905327 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp"] Nov 03 04:04:57.942949 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.942899 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-metrics-certs\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:57.943166 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.942954 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kclw5\" (UniqueName: \"kubernetes.io/projected/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-kube-api-access-kclw5\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:57.943166 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.943092 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:57.943304 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:57.943207 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.044965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.044877 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-metrics-certs\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.044965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.044919 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kclw5\" (UniqueName: \"kubernetes.io/projected/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-kube-api-access-kclw5\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.045177 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.044995 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.045177 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.045074 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.045835 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.045804 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.048231 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.048205 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-metrics-certs\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.048586 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.048322 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.146934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.146902 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kclw5\" (UniqueName: \"kubernetes.io/projected/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-kube-api-access-kclw5\") pod \"maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.412325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.412295 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:04:58.790975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:58.790944 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp"] Nov 03 04:04:58.791939 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:04:58.791910 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea672fa7_1b0d_49ad_8ecb_5ec9b98dc77a.slice/crio-7fd5df7776fbeb8a2c3f1a0c8864295dc6684bd45658da2539ed389e7e272afc WatchSource:0}: Error finding container 7fd5df7776fbeb8a2c3f1a0c8864295dc6684bd45658da2539ed389e7e272afc: Status 404 returned error can't find the container with id 7fd5df7776fbeb8a2c3f1a0c8864295dc6684bd45658da2539ed389e7e272afc Nov 03 04:04:59.623167 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:04:59.623130 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" event={"ID":"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a","Type":"ContainerStarted","Data":"7fd5df7776fbeb8a2c3f1a0c8864295dc6684bd45658da2539ed389e7e272afc"} Nov 03 04:05:00.012700 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:05:00.012611 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:05:00.630643 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:00.630605 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" event={"ID":"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a","Type":"ContainerStarted","Data":"688fca5c403a04c046c1b0b9d8824098ad82b198d7a1bc6545e490c7eb9ce5fe"} Nov 03 04:05:00.631050 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:00.630663 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:05:00.751382 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:00.751328 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" podStartSLOduration=2.858383199 podStartE2EDuration="3.751314807s" totalImagesPullingTime="892.931608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:05:00.750773519 +0000 UTC m=+3835.781372058" watchObservedRunningTime="2026-11-03 04:05:00.751314807 +0000 UTC m=+3835.781913344" Nov 03 04:05:06.642730 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:06.642698 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:05:30.011365 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:05:30.011321 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:05:57.560968 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.560919 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474"] Nov 03 04:05:57.561626 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.561199 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" podUID="7f5e5bd9-7556-4b3d-91b0-5463c77d0016" containerName="maas-api" containerID="cri-o://02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803" gracePeriod=30 Nov 03 04:05:57.622374 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:05:57.622336 5117 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5e5bd9_7556_4b3d_91b0_5463c77d0016.slice/crio-conmon-02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803.scope\": RecentStats: unable to find data in memory cache]" Nov 03 04:05:57.622609 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:05:57.622580 5117 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5e5bd9_7556_4b3d_91b0_5463c77d0016.slice/crio-conmon-02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803.scope\": RecentStats: unable to find data in memory cache]" Nov 03 04:05:57.875521 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.875493 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:05:57.973428 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.973394 5117 generic.go:356] "Generic (PLEG): container finished" podID="7f5e5bd9-7556-4b3d-91b0-5463c77d0016" containerID="02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803" exitCode=0 Nov 03 04:05:57.973658 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.973479 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" Nov 03 04:05:57.973658 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.973492 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" event={"ID":"7f5e5bd9-7556-4b3d-91b0-5463c77d0016","Type":"ContainerDied","Data":"02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803"} Nov 03 04:05:57.973658 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.973562 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474" event={"ID":"7f5e5bd9-7556-4b3d-91b0-5463c77d0016","Type":"ContainerDied","Data":"b491522d56dc49dc9c864510ade00a7fbee8db141529b51331446b9a6cf66f45"} Nov 03 04:05:57.973658 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.973588 5117 scope.go:122] "RemoveContainer" containerID="02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803" Nov 03 04:05:57.983980 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.983959 5117 scope.go:122] "RemoveContainer" containerID="02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803" Nov 03 04:05:57.984335 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:05:57.984312 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803\": container with ID starting with 02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803 not found: ID does not exist" containerID="02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803" Nov 03 04:05:57.984432 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.984350 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803"} err="failed to get container status \"02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803\": rpc error: code = NotFound desc = could not find container \"02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803\": container with ID starting with 02ba1eddfd872110eb7365da14a253d4a38371d84016bbdab19fd54bbadbe803 not found: ID does not exist" Nov 03 04:05:57.992752 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.992727 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-metrics-certs\") pod \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " Nov 03 04:05:57.992884 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.992822 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-kube-api-access-wtpnh\" (UniqueName: \"kubernetes.io/projected/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-kube-api-access-wtpnh\") pod \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " Nov 03 04:05:57.992962 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.992935 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls\") pod \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " Nov 03 04:05:57.992962 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.992954 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-openshift-service-ca\") pod \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\" (UID: \"7f5e5bd9-7556-4b3d-91b0-5463c77d0016\") " Nov 03 04:05:57.993374 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.993347 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-openshift-service-ca" pod "7f5e5bd9-7556-4b3d-91b0-5463c77d0016" (UID: "7f5e5bd9-7556-4b3d-91b0-5463c77d0016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:05:57.995313 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.995282 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-metrics-certs" pod "7f5e5bd9-7556-4b3d-91b0-5463c77d0016" (UID: "7f5e5bd9-7556-4b3d-91b0-5463c77d0016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:05:57.995706 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.995674 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls" pod "7f5e5bd9-7556-4b3d-91b0-5463c77d0016" (UID: "7f5e5bd9-7556-4b3d-91b0-5463c77d0016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:05:57.995828 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:57.995761 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-kube-api-access-wtpnh" pod "7f5e5bd9-7556-4b3d-91b0-5463c77d0016" (UID: "7f5e5bd9-7556-4b3d-91b0-5463c77d0016"). InnerVolumeSpecName "kube-api-access-wtpnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:05:58.094813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:58.094730 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:58.094813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:58.094761 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:58.094813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:58.094773 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:58.094813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:58.094782 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpnh\" (UniqueName: \"kubernetes.io/projected/7f5e5bd9-7556-4b3d-91b0-5463c77d0016-kube-api-access-wtpnh\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:05:58.398111 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:58.398030 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474"] Nov 03 04:05:58.460482 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:58.459088 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0afad-546dbfc74-6h474"] Nov 03 04:05:59.091461 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:05:59.091417 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5e5bd9-7556-4b3d-91b0-5463c77d0016" path="/var/lib/kubelet/pods/7f5e5bd9-7556-4b3d-91b0-5463c77d0016/volumes" Nov 03 04:06:00.010468 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:06:00.010403 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:06:07.706541 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:07.706507 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:06:07.712213 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:07.712187 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:06:07.717150 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:07.717116 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:06:07.722606 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:07.722587 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:06:30.011655 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:06:30.011615 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:06:40.173157 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.173117 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8"] Nov 03 04:06:40.173743 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.173338 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" podUID="714e799a-0d64-4598-aaa1-92ff5a63ff91" containerName="maas-api" containerID="cri-o://8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd" gracePeriod=30 Nov 03 04:06:40.443343 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.443277 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:06:40.541943 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.541907 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls\") pod \"714e799a-0d64-4598-aaa1-92ff5a63ff91\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " Nov 03 04:06:40.542120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.542082 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714e799a-0d64-4598-aaa1-92ff5a63ff91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714e799a-0d64-4598-aaa1-92ff5a63ff91-openshift-service-ca\") pod \"714e799a-0d64-4598-aaa1-92ff5a63ff91\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " Nov 03 04:06:40.542200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.542183 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-metrics-certs\") pod \"714e799a-0d64-4598-aaa1-92ff5a63ff91\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " Nov 03 04:06:40.542247 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.542218 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714e799a-0d64-4598-aaa1-92ff5a63ff91-kube-api-access-qkkk4\" (UniqueName: \"kubernetes.io/projected/714e799a-0d64-4598-aaa1-92ff5a63ff91-kube-api-access-qkkk4\") pod \"714e799a-0d64-4598-aaa1-92ff5a63ff91\" (UID: \"714e799a-0d64-4598-aaa1-92ff5a63ff91\") " Nov 03 04:06:40.542546 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.542515 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714e799a-0d64-4598-aaa1-92ff5a63ff91-openshift-service-ca" pod "714e799a-0d64-4598-aaa1-92ff5a63ff91" (UID: "714e799a-0d64-4598-aaa1-92ff5a63ff91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:06:40.542666 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.542632 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714e799a-0d64-4598-aaa1-92ff5a63ff91-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:40.544793 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.544765 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls" pod "714e799a-0d64-4598-aaa1-92ff5a63ff91" (UID: "714e799a-0d64-4598-aaa1-92ff5a63ff91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:06:40.544990 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.544970 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-metrics-certs" pod "714e799a-0d64-4598-aaa1-92ff5a63ff91" (UID: "714e799a-0d64-4598-aaa1-92ff5a63ff91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:06:40.545061 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.545029 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714e799a-0d64-4598-aaa1-92ff5a63ff91-kube-api-access-qkkk4" pod "714e799a-0d64-4598-aaa1-92ff5a63ff91" (UID: "714e799a-0d64-4598-aaa1-92ff5a63ff91"). InnerVolumeSpecName "kube-api-access-qkkk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:06:40.643437 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.643408 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:40.643437 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.643434 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/714e799a-0d64-4598-aaa1-92ff5a63ff91-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:40.643437 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:40.643459 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkk4\" (UniqueName: \"kubernetes.io/projected/714e799a-0d64-4598-aaa1-92ff5a63ff91-kube-api-access-qkkk4\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:06:41.244838 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.244801 5117 generic.go:356] "Generic (PLEG): container finished" podID="714e799a-0d64-4598-aaa1-92ff5a63ff91" containerID="8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd" exitCode=0 Nov 03 04:06:41.245332 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.244854 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" Nov 03 04:06:41.245332 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.244861 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" event={"ID":"714e799a-0d64-4598-aaa1-92ff5a63ff91","Type":"ContainerDied","Data":"8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd"} Nov 03 04:06:41.245332 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.244914 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8" event={"ID":"714e799a-0d64-4598-aaa1-92ff5a63ff91","Type":"ContainerDied","Data":"febdf900524c2c3549527fcf41065dff011156151e4782648fed3dcf53c10c80"} Nov 03 04:06:41.245332 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.244946 5117 scope.go:122] "RemoveContainer" containerID="8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd" Nov 03 04:06:41.266597 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.266569 5117 scope.go:122] "RemoveContainer" containerID="8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd" Nov 03 04:06:41.267008 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:06:41.266985 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd\": container with ID starting with 8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd not found: ID does not exist" containerID="8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd" Nov 03 04:06:41.267108 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.267014 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd"} err="failed to get container status \"8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd\": rpc error: code = NotFound desc = could not find container \"8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd\": container with ID starting with 8a9087bcbdc58b87efdc6007d1086934118a88ba5ae4abaa5d42a104de8a69dd not found: ID does not exist" Nov 03 04:06:41.412665 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.412562 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8"] Nov 03 04:06:41.463569 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:41.463540 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-03da9e-7cc4f59ffc-qsvf8"] Nov 03 04:06:43.090177 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:43.090131 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714e799a-0d64-4598-aaa1-92ff5a63ff91" path="/var/lib/kubelet/pods/714e799a-0d64-4598-aaa1-92ff5a63ff91/volumes" Nov 03 04:06:47.656400 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.656356 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp"] Nov 03 04:06:47.656949 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.656897 5117 cpu_manager.go:414] "containerMap: removing container" podUID="714e799a-0d64-4598-aaa1-92ff5a63ff91" containerName="maas-api" Nov 03 04:06:47.656949 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.656910 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714e799a-0d64-4598-aaa1-92ff5a63ff91" containerName="maas-api" Nov 03 04:06:47.656949 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.656938 5117 cpu_manager.go:414] "containerMap: removing container" podUID="7f5e5bd9-7556-4b3d-91b0-5463c77d0016" containerName="maas-api" Nov 03 04:06:47.656949 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.656943 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5e5bd9-7556-4b3d-91b0-5463c77d0016" containerName="maas-api" Nov 03 04:06:47.657084 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.657023 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5e5bd9-7556-4b3d-91b0-5463c77d0016" containerName="maas-api" Nov 03 04:06:47.657084 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.657037 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="714e799a-0d64-4598-aaa1-92ff5a63ff91" containerName="maas-api" Nov 03 04:06:47.661752 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.661732 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.769227 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.769191 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp"] Nov 03 04:06:47.798616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.798590 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gfd7r\"" Nov 03 04:06:47.832186 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832143 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832186 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832181 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9smh\" (UniqueName: \"kubernetes.io/projected/21acc925-025e-4dd2-bd7e-3314f7337a45-kube-api-access-r9smh\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832326 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832360 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832579 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832473 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832644 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832575 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832700 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832650 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21acc925-025e-4dd2-bd7e-3314f7337a45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832751 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832731 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.832818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.832798 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934025 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.933936 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934025 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.933992 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934256 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934036 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9smh\" (UniqueName: \"kubernetes.io/projected/21acc925-025e-4dd2-bd7e-3314f7337a45-kube-api-access-r9smh\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934256 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934083 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934256 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934108 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934256 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934173 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934256 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934239 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934307 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21acc925-025e-4dd2-bd7e-3314f7337a45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934351 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934547 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934571 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934591 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.934810 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.934745 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.935208 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.935186 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21acc925-025e-4dd2-bd7e-3314f7337a45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.936693 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.936666 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.939326 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.939299 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.996356 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.996323 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21acc925-025e-4dd2-bd7e-3314f7337a45-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:47.996581 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:47.996388 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9smh\" (UniqueName: \"kubernetes.io/projected/21acc925-025e-4dd2-bd7e-3314f7337a45-kube-api-access-r9smh\") pod \"maas-default-gateway-openshift-default-76f4876667-lbdzp\" (UID: \"21acc925-025e-4dd2-bd7e-3314f7337a45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:48.275433 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:48.275352 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:48.709328 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:48.709296 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp"] Nov 03 04:06:48.710233 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:06:48.710207 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21acc925_025e_4dd2_bd7e_3314f7337a45.slice/crio-ab3d43379b942273210784a4c5ee11b664eaebdbf55019cb233c9466105ea217 WatchSource:0}: Error finding container ab3d43379b942273210784a4c5ee11b664eaebdbf55019cb233c9466105ea217: Status 404 returned error can't find the container with id ab3d43379b942273210784a4c5ee11b664eaebdbf55019cb233c9466105ea217 Nov 03 04:06:48.712761 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:48.712730 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:06:48.712850 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:48.712796 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:06:48.712850 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:48.712845 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:06:49.298278 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:49.298242 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" event={"ID":"21acc925-025e-4dd2-bd7e-3314f7337a45","Type":"ContainerStarted","Data":"2df503759b275fe6ee6f05cc3cac40ca94c4187a1b4148eb64b2dec7119ffc43"} Nov 03 04:06:49.298504 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:49.298314 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" event={"ID":"21acc925-025e-4dd2-bd7e-3314f7337a45","Type":"ContainerStarted","Data":"ab3d43379b942273210784a4c5ee11b664eaebdbf55019cb233c9466105ea217"} Nov 03 04:06:49.426390 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:49.426328 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podStartSLOduration=2.426312924 podStartE2EDuration="2.426312924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:06:49.422770729 +0000 UTC m=+3944.453369269" watchObservedRunningTime="2026-11-03 04:06:49.426312924 +0000 UTC m=+3944.456911461" Nov 03 04:06:50.276401 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:50.276364 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:50.278302 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:50.278273 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 04:06:50.278434 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:50.278324 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 04:06:51.276701 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:51.276665 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 04:06:51.277260 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:51.276743 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 04:06:52.275826 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:52.275780 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 04:06:52.276048 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:52.275847 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 04:06:53.276274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:53.276239 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 04:06:53.276702 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:53.276286 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 04:06:54.276110 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:54.276066 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 04:06:54.276286 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:54.276143 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 04:06:55.276563 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:55.276518 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 04:06:55.277070 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:55.276576 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 04:06:56.276322 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:56.276289 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 04:06:56.276574 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:56.276356 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 04:06:57.428548 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:57.428505 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:57.428973 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:57.428949 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:06:57.429927 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:06:57.429901 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" Nov 03 04:07:00.010612 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:07:00.010567 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:07:30.011235 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:07:30.011193 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:07:48.129644 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.129568 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw"] Nov 03 04:07:48.133507 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.133488 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.224836 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.224803 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.224978 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.224852 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.224978 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.224926 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.224978 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.224969 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.225126 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.224987 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8mdw\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-kube-api-access-h8mdw\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.225126 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.225015 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.225232 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.225199 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.225283 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.225265 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.225336 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.225321 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.326784 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.326745 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.326965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.326791 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.326965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.326818 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.326965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.326875 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.326965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.326902 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8mdw\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-kube-api-access-h8mdw\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.326965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.326933 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.326981 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.327031 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.327082 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327407 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.327377 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.327512 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327606 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.327582 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327653 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.327607 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.327890 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.327863 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.329336 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.329312 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.329557 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.329540 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.338748 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.338726 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw"] Nov 03 04:07:48.388388 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.388317 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.388555 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.388514 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8mdw\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-kube-api-access-h8mdw\") pod \"maas-default-gateway-openshift-default-76f4876667-lx4vw\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.446645 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.446614 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:48.933778 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.933751 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw"] Nov 03 04:07:48.934202 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:07:48.934180 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e1aa63_6b48_40dd_ad68_ef76b84e6374.slice/crio-4bd5df4a8a6a6b49d4b0249adda2ac127f393c112c2b5c84df66b00fea65bc6a WatchSource:0}: Error finding container 4bd5df4a8a6a6b49d4b0249adda2ac127f393c112c2b5c84df66b00fea65bc6a: Status 404 returned error can't find the container with id 4bd5df4a8a6a6b49d4b0249adda2ac127f393c112c2b5c84df66b00fea65bc6a Nov 03 04:07:48.936544 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.936512 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:07:48.936652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.936592 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:07:48.936652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:48.936639 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:07:49.691466 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:49.691407 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" event={"ID":"11e1aa63-6b48-40dd-ad68-ef76b84e6374","Type":"ContainerStarted","Data":"2f333aa39d8d83e2972d1464a9cd82e82554990d8da684d882a81405a66b91ef"} Nov 03 04:07:49.691898 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:49.691496 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" event={"ID":"11e1aa63-6b48-40dd-ad68-ef76b84e6374","Type":"ContainerStarted","Data":"4bd5df4a8a6a6b49d4b0249adda2ac127f393c112c2b5c84df66b00fea65bc6a"} Nov 03 04:07:49.865212 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:49.865147 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podStartSLOduration=1.865133588 podStartE2EDuration="1.865133588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:07:49.863422029 +0000 UTC m=+4004.894020568" watchObservedRunningTime="2026-11-03 04:07:49.865133588 +0000 UTC m=+4004.895732125" Nov 03 04:07:50.447244 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.447205 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:50.449151 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.449120 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 04:07:50.449303 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.449181 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 04:07:50.555518 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.555481 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp"] Nov 03 04:07:50.555772 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.555742 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" podUID="ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" containerName="maas-api" containerID="cri-o://688fca5c403a04c046c1b0b9d8824098ad82b198d7a1bc6545e490c7eb9ce5fe" gracePeriod=30 Nov 03 04:07:50.712545 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.700642 5117 generic.go:356] "Generic (PLEG): container finished" podID="ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" containerID="688fca5c403a04c046c1b0b9d8824098ad82b198d7a1bc6545e490c7eb9ce5fe" exitCode=0 Nov 03 04:07:50.712545 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.701652 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" event={"ID":"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a","Type":"ContainerDied","Data":"688fca5c403a04c046c1b0b9d8824098ad82b198d7a1bc6545e490c7eb9ce5fe"} Nov 03 04:07:50.908558 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.908057 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:07:50.957297 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.957250 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-maas-api-tls\") pod \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " Nov 03 04:07:50.957516 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.957331 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-openshift-service-ca\") pod \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " Nov 03 04:07:50.957578 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.957532 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-kube-api-access-kclw5\" (UniqueName: \"kubernetes.io/projected/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-kube-api-access-kclw5\") pod \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " Nov 03 04:07:50.957616 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.957586 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-metrics-certs\") pod \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\" (UID: \"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a\") " Nov 03 04:07:50.958226 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.958064 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-openshift-service-ca" pod "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" (UID: "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:07:50.961021 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.960969 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-maas-api-tls" pod "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" (UID: "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:07:50.961144 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.961045 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-kube-api-access-kclw5" pod "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" (UID: "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a"). InnerVolumeSpecName "kube-api-access-kclw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:07:50.962661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:50.962592 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-metrics-certs" pod "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" (UID: "ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:07:51.061331 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.059864 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kclw5\" (UniqueName: \"kubernetes.io/projected/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-kube-api-access-kclw5\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:51.061331 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.059905 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:51.061331 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.059922 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:51.061331 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.059949 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:07:51.447250 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.447216 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 04:07:51.447456 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.447284 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 04:07:51.731801 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.731721 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" Nov 03 04:07:51.732185 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.731873 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp" event={"ID":"ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a","Type":"ContainerDied","Data":"7fd5df7776fbeb8a2c3f1a0c8864295dc6684bd45658da2539ed389e7e272afc"} Nov 03 04:07:51.732185 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.731931 5117 scope.go:122] "RemoveContainer" containerID="688fca5c403a04c046c1b0b9d8824098ad82b198d7a1bc6545e490c7eb9ce5fe" Nov 03 04:07:51.911470 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.899793 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp"] Nov 03 04:07:51.925655 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:51.925615 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-90139c-5675c7888-ps6dp"] Nov 03 04:07:52.447204 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:52.447168 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 04:07:52.447406 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:52.447220 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 04:07:53.089730 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:53.089696 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" path="/var/lib/kubelet/pods/ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a/volumes" Nov 03 04:07:53.447368 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:53.447285 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 04:07:53.447368 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:53.447350 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 04:07:54.447103 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:54.447071 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 04:07:54.447641 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:54.447591 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 04:07:55.447336 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:55.447303 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 04:07:55.447849 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:55.447364 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 04:07:56.447008 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:56.446972 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 04:07:56.447227 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:56.447045 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 04:07:57.981958 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:57.981919 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:57.982801 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:57.982774 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:07:57.984035 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:07:57.984013 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:08:00.011531 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:08:00.011438 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:08:30.013474 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:08:30.013407 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:08:30.976265 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:30.976231 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x"] Nov 03 04:08:30.976508 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:30.976460 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" podUID="8801da75-7d4d-463b-807b-5ba26b9e34fa" containerName="maas-api" containerID="cri-o://ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9" gracePeriod=30 Nov 03 04:08:31.264279 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.264246 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:08:31.362565 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.362533 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls\") pod \"8801da75-7d4d-463b-807b-5ba26b9e34fa\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " Nov 03 04:08:31.362764 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.362577 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8801da75-7d4d-463b-807b-5ba26b9e34fa-kube-api-access-vqsx8\" (UniqueName: \"kubernetes.io/projected/8801da75-7d4d-463b-807b-5ba26b9e34fa-kube-api-access-vqsx8\") pod \"8801da75-7d4d-463b-807b-5ba26b9e34fa\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " Nov 03 04:08:31.362764 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.362621 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-metrics-certs\") pod \"8801da75-7d4d-463b-807b-5ba26b9e34fa\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " Nov 03 04:08:31.362878 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.362855 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8801da75-7d4d-463b-807b-5ba26b9e34fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8801da75-7d4d-463b-807b-5ba26b9e34fa-openshift-service-ca\") pod \"8801da75-7d4d-463b-807b-5ba26b9e34fa\" (UID: \"8801da75-7d4d-463b-807b-5ba26b9e34fa\") " Nov 03 04:08:31.363479 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.363435 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8801da75-7d4d-463b-807b-5ba26b9e34fa-openshift-service-ca" pod "8801da75-7d4d-463b-807b-5ba26b9e34fa" (UID: "8801da75-7d4d-463b-807b-5ba26b9e34fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:08:31.371169 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.371107 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-metrics-certs" pod "8801da75-7d4d-463b-807b-5ba26b9e34fa" (UID: "8801da75-7d4d-463b-807b-5ba26b9e34fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:08:31.374665 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.374566 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls" pod "8801da75-7d4d-463b-807b-5ba26b9e34fa" (UID: "8801da75-7d4d-463b-807b-5ba26b9e34fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:08:31.374665 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.374627 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8801da75-7d4d-463b-807b-5ba26b9e34fa-kube-api-access-vqsx8" pod "8801da75-7d4d-463b-807b-5ba26b9e34fa" (UID: "8801da75-7d4d-463b-807b-5ba26b9e34fa"). InnerVolumeSpecName "kube-api-access-vqsx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:08:31.464392 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.464364 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8801da75-7d4d-463b-807b-5ba26b9e34fa-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:31.464392 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.464391 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:31.464602 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.464401 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqsx8\" (UniqueName: \"kubernetes.io/projected/8801da75-7d4d-463b-807b-5ba26b9e34fa-kube-api-access-vqsx8\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:31.464602 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.464411 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8801da75-7d4d-463b-807b-5ba26b9e34fa-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:08:31.994752 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.994715 5117 generic.go:356] "Generic (PLEG): container finished" podID="8801da75-7d4d-463b-807b-5ba26b9e34fa" containerID="ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9" exitCode=0 Nov 03 04:08:31.994932 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.994778 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" Nov 03 04:08:31.994932 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.994801 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" event={"ID":"8801da75-7d4d-463b-807b-5ba26b9e34fa","Type":"ContainerDied","Data":"ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9"} Nov 03 04:08:31.994932 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.994865 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x" event={"ID":"8801da75-7d4d-463b-807b-5ba26b9e34fa","Type":"ContainerDied","Data":"b870c8ffc7cf1b2bc8647175d91081d1181e8c4407c59c1109632f9e16e56e3c"} Nov 03 04:08:31.994932 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:31.994889 5117 scope.go:122] "RemoveContainer" containerID="ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9" Nov 03 04:08:32.005859 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:32.005838 5117 scope.go:122] "RemoveContainer" containerID="ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9" Nov 03 04:08:32.006135 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:08:32.006115 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9\": container with ID starting with ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9 not found: ID does not exist" containerID="ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9" Nov 03 04:08:32.006184 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:32.006145 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9"} err="failed to get container status \"ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9\": rpc error: code = NotFound desc = could not find container \"ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9\": container with ID starting with ee31b1ed76051c2d1196fecc4893833bff851e1abd1a3831e828f09e3757b7a9 not found: ID does not exist" Nov 03 04:08:32.167623 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:32.167584 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x"] Nov 03 04:08:32.218782 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:32.218748 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c832d-64848994cf-kkn5x"] Nov 03 04:08:33.091422 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.091389 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8801da75-7d4d-463b-807b-5ba26b9e34fa" path="/var/lib/kubelet/pods/8801da75-7d4d-463b-807b-5ba26b9e34fa/volumes" Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.512715 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5"] Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.513630 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8801da75-7d4d-463b-807b-5ba26b9e34fa" containerName="maas-api" Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.513649 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8801da75-7d4d-463b-807b-5ba26b9e34fa" containerName="maas-api" Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.513674 5117 cpu_manager.go:414] "containerMap: removing container" podUID="ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" containerName="maas-api" Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.513683 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" containerName="maas-api" Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.513817 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8801da75-7d4d-463b-807b-5ba26b9e34fa" containerName="maas-api" Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.513834 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea672fa7-1b0d-49ad-8ecb-5ec9b98dc77a" containerName="maas-api" Nov 03 04:08:33.522560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.522230 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.592140 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.592090 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5"] Nov 03 04:08:33.689969 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.689912 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15dddef-9e9b-4662-a65f-614203a4e981-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690188 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690074 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15dddef-9e9b-4662-a65f-614203a4e981-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690257 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690199 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690318 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690254 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690318 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690289 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690530 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690504 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690641 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690596 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15dddef-9e9b-4662-a65f-614203a4e981-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690641 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690621 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.690724 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.690641 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljtl\" (UniqueName: \"kubernetes.io/projected/b15dddef-9e9b-4662-a65f-614203a4e981-kube-api-access-8ljtl\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792215 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792106 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792215 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792160 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792215 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792183 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792229 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792278 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15dddef-9e9b-4662-a65f-614203a4e981-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792296 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792320 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljtl\" (UniqueName: \"kubernetes.io/projected/b15dddef-9e9b-4662-a65f-614203a4e981-kube-api-access-8ljtl\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792358 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15dddef-9e9b-4662-a65f-614203a4e981-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792416 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15dddef-9e9b-4662-a65f-614203a4e981-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792838 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792654 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.792985 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.792961 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.793064 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.793019 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.793155 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.793134 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.793222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.793161 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b15dddef-9e9b-4662-a65f-614203a4e981-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.795665 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.795633 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b15dddef-9e9b-4662-a65f-614203a4e981-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.795845 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.795828 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b15dddef-9e9b-4662-a65f-614203a4e981-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.854391 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.854351 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b15dddef-9e9b-4662-a65f-614203a4e981-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:33.854625 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:33.854600 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljtl\" (UniqueName: \"kubernetes.io/projected/b15dddef-9e9b-4662-a65f-614203a4e981-kube-api-access-8ljtl\") pod \"maas-default-gateway-openshift-default-76f4876667-9znf5\" (UID: \"b15dddef-9e9b-4662-a65f-614203a4e981\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:34.144209 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:34.144163 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:34.579854 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:08:34.579812 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15dddef_9e9b_4662_a65f_614203a4e981.slice/crio-b127311a71d267d21ee9972c61cf8733d3d82c52c9ec099ab778e5590568860f WatchSource:0}: Error finding container b127311a71d267d21ee9972c61cf8733d3d82c52c9ec099ab778e5590568860f: Status 404 returned error can't find the container with id b127311a71d267d21ee9972c61cf8733d3d82c52c9ec099ab778e5590568860f Nov 03 04:08:34.580255 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:34.580228 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5"] Nov 03 04:08:34.583132 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:34.583089 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:08:34.583247 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:34.583189 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:08:34.583247 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:34.583234 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:08:35.017120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:35.017027 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" event={"ID":"b15dddef-9e9b-4662-a65f-614203a4e981","Type":"ContainerStarted","Data":"22ceb055f63ccdd8a122b764b3243f50e6d281bb44bc9003f30a3c9cd294a25a"} Nov 03 04:08:35.017120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:35.017079 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" event={"ID":"b15dddef-9e9b-4662-a65f-614203a4e981","Type":"ContainerStarted","Data":"b127311a71d267d21ee9972c61cf8733d3d82c52c9ec099ab778e5590568860f"} Nov 03 04:08:35.145062 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:35.145027 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:35.146532 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:35.146507 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:35.146664 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:35.146553 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:35.193060 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:35.193007 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podStartSLOduration=2.192994296 podStartE2EDuration="2.192994296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:35.190263491 +0000 UTC m=+4050.220862030" watchObservedRunningTime="2026-11-03 04:08:35.192994296 +0000 UTC m=+4050.223592834" Nov 03 04:08:36.145555 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:36.145512 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:36.146102 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:36.145584 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:37.144967 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:37.144922 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:37.145146 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:37.144986 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:38.145267 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:38.145234 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:38.145798 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:38.145293 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:39.144820 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:39.144779 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:39.145005 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:39.144851 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:40.145244 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:40.145201 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:40.145690 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:40.145268 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:41.145084 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:41.145047 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:41.145258 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:41.145128 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:42.144914 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:42.144876 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 04:08:42.145294 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:42.144946 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 04:08:43.568661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.568618 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn"] Nov 03 04:08:43.573873 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.573846 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:43.622391 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.622326 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:43.622682 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.622653 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:43.623588 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.623566 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" Nov 03 04:08:43.629366 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.629335 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd4d2\" (UniqueName: \"kubernetes.io/projected/ab266756-b7f2-48b0-a26e-9be7c2deb5df-kube-api-access-cd4d2\") pod \"payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:43.629689 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.629663 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab266756-b7f2-48b0-a26e-9be7c2deb5df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:43.633976 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.633953 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn"] Nov 03 04:08:43.679753 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.679716 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-934bd4\"" Nov 03 04:08:43.679921 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.679773 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-934bd4-dockercfg-2dvms\"" Nov 03 04:08:43.730710 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.730678 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd4d2\" (UniqueName: \"kubernetes.io/projected/ab266756-b7f2-48b0-a26e-9be7c2deb5df-kube-api-access-cd4d2\") pod \"payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:43.730913 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.730846 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab266756-b7f2-48b0-a26e-9be7c2deb5df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:43.732021 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.731978 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab266756-b7f2-48b0-a26e-9be7c2deb5df-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:43.793171 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.793145 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd4d2\" (UniqueName: \"kubernetes.io/projected/ab266756-b7f2-48b0-a26e-9be7c2deb5df-kube-api-access-cd4d2\") pod \"payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:43.895030 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:43.894957 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:44.273521 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:44.273491 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn"] Nov 03 04:08:44.274008 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:08:44.273976 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab266756_b7f2_48b0_a26e_9be7c2deb5df.slice/crio-38ca00b1b77cdc3087e2777acccedd87a9757530303213cb75b44f20983047fb WatchSource:0}: Error finding container 38ca00b1b77cdc3087e2777acccedd87a9757530303213cb75b44f20983047fb: Status 404 returned error can't find the container with id 38ca00b1b77cdc3087e2777acccedd87a9757530303213cb75b44f20983047fb Nov 03 04:08:44.276009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:44.275985 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 04:08:45.092842 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:45.092799 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" event={"ID":"ab266756-b7f2-48b0-a26e-9be7c2deb5df","Type":"ContainerStarted","Data":"38ca00b1b77cdc3087e2777acccedd87a9757530303213cb75b44f20983047fb"} Nov 03 04:08:47.116415 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:47.116377 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" event={"ID":"ab266756-b7f2-48b0-a26e-9be7c2deb5df","Type":"ContainerStarted","Data":"444f2bdd82b0153519ed60d6e87bad13fe6ceba95e46b2b545d9a8f8327dc292"} Nov 03 04:08:47.116741 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:47.116457 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:08:47.237982 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:47.237931 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" podStartSLOduration=1.489971212 podStartE2EDuration="4.23791515s" totalImagesPullingTime="2.747943938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:47.233802432 +0000 UTC m=+4062.264400971" watchObservedRunningTime="2026-11-03 04:08:47.23791515 +0000 UTC m=+4062.268513688" Nov 03 04:08:58.127110 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:08:58.127081 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:09:00.011495 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:09:00.011460 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:09:22.157712 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.157677 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj"] Nov 03 04:09:22.166356 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.166332 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.221259 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.221219 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj"] Nov 03 04:09:22.268438 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.268402 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ed4ecc-serving-cert\"" Nov 03 04:09:22.268876 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.268846 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljkc\" (UniqueName: \"kubernetes.io/projected/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-kube-api-access-dljkc\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.268922 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.268911 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.269023 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.269004 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.269161 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.269148 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-metrics-certs\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.369890 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.369850 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.370096 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.369966 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-metrics-certs\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.370096 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.370007 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljkc\" (UniqueName: \"kubernetes.io/projected/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-kube-api-access-dljkc\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.370096 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.370029 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.370260 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:09:22.370127 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-serving-cert: secret "maas-api-e2e-inf-a-ed4ecc-serving-cert" not found Nov 03 04:09:22.370260 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:09:22.370186 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls podName:8d8b4818-a5e0-4b3f-b82f-479d243dd14c nodeName:}" failed. No retries permitted until 2026-11-03 04:09:22.870168426 +0000 UTC m=+4097.900766959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls") pod "maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" (UID: "8d8b4818-a5e0-4b3f-b82f-479d243dd14c") : secret "maas-api-e2e-inf-a-ed4ecc-serving-cert" not found Nov 03 04:09:22.370615 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.370591 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.372622 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.372600 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-metrics-certs\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.434528 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.434425 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljkc\" (UniqueName: \"kubernetes.io/projected/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-kube-api-access-dljkc\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.876112 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.876076 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:22.878668 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:22.878644 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:23.080231 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:23.080200 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:23.460756 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:23.460723 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj"] Nov 03 04:09:23.461330 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:09:23.461306 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8b4818_a5e0_4b3f_b82f_479d243dd14c.slice/crio-02f5663f69a74e3d613b9a5e928284627107379ed78d415b86601c3a15de1917 WatchSource:0}: Error finding container 02f5663f69a74e3d613b9a5e928284627107379ed78d415b86601c3a15de1917: Status 404 returned error can't find the container with id 02f5663f69a74e3d613b9a5e928284627107379ed78d415b86601c3a15de1917 Nov 03 04:09:24.343687 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:24.343639 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" event={"ID":"8d8b4818-a5e0-4b3f-b82f-479d243dd14c","Type":"ContainerStarted","Data":"02f5663f69a74e3d613b9a5e928284627107379ed78d415b86601c3a15de1917"} Nov 03 04:09:25.351797 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:25.351758 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" event={"ID":"8d8b4818-a5e0-4b3f-b82f-479d243dd14c","Type":"ContainerStarted","Data":"3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715"} Nov 03 04:09:25.352218 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:25.351817 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:25.473618 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:25.473559 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" podStartSLOduration=2.528213058 podStartE2EDuration="3.473545456s" totalImagesPullingTime="945.332398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:09:25.471186649 +0000 UTC m=+4100.501785188" watchObservedRunningTime="2026-11-03 04:09:25.473545456 +0000 UTC m=+4100.504143994" Nov 03 04:09:27.077317 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.077282 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz"] Nov 03 04:09:27.081366 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.081347 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.187187 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.187150 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-214eab-serving-cert\"" Nov 03 04:09:27.190497 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.190466 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz"] Nov 03 04:09:27.230466 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.230414 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc070a2e-08b9-4f80-8454-4a595fa55fa9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.230659 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.230622 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-metrics-certs\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.230793 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.230776 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-maas-api-tls\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.230861 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.230848 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p62r\" (UniqueName: \"kubernetes.io/projected/dc070a2e-08b9-4f80-8454-4a595fa55fa9-kube-api-access-5p62r\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.332362 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.332274 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc070a2e-08b9-4f80-8454-4a595fa55fa9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.332362 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.332351 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-metrics-certs\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.332636 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.332490 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-maas-api-tls\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.332636 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.332547 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p62r\" (UniqueName: \"kubernetes.io/projected/dc070a2e-08b9-4f80-8454-4a595fa55fa9-kube-api-access-5p62r\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.333160 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.333099 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc070a2e-08b9-4f80-8454-4a595fa55fa9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.335136 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.335114 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-metrics-certs\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.335222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.335175 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-maas-api-tls\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.397128 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.397091 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p62r\" (UniqueName: \"kubernetes.io/projected/dc070a2e-08b9-4f80-8454-4a595fa55fa9-kube-api-access-5p62r\") pod \"maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.397683 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.397651 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:27.785485 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:27.783976 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz"] Nov 03 04:09:28.374055 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:28.374015 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" event={"ID":"dc070a2e-08b9-4f80-8454-4a595fa55fa9","Type":"ContainerStarted","Data":"215b6d8354ec28020f8268f7f248e3e1b34024a91e078802286f937cc0784d34"} Nov 03 04:09:29.383174 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:29.383135 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" event={"ID":"dc070a2e-08b9-4f80-8454-4a595fa55fa9","Type":"ContainerStarted","Data":"9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a"} Nov 03 04:09:29.383650 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:29.383240 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:29.507222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:29.507170 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" podStartSLOduration=1.712407039 podStartE2EDuration="2.507151752s" totalImagesPullingTime="794.744713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:09:29.5048459 +0000 UTC m=+4104.535444437" watchObservedRunningTime="2026-11-03 04:09:29.507151752 +0000 UTC m=+4104.537750289" Nov 03 04:09:30.012226 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:09:30.012188 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:09:31.366020 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:31.365993 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:09:35.394955 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:35.394920 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:09:36.861382 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.861339 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5"] Nov 03 04:09:36.865641 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.865616 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:36.920417 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.920385 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c4fa79-serving-cert\"" Nov 03 04:09:36.939200 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.939157 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3759f5-6d27-49c7-801d-11ca49d6d787-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:36.939401 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.939302 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:36.939401 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.939339 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-metrics-certs\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:36.939568 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.939465 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chjn\" (UniqueName: \"kubernetes.io/projected/ab3759f5-6d27-49c7-801d-11ca49d6d787-kube-api-access-8chjn\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:36.974016 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:36.973984 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5"] Nov 03 04:09:37.040822 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.040788 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3759f5-6d27-49c7-801d-11ca49d6d787-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.041009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.040884 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.041009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.040919 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-metrics-certs\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.041141 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:09:37.041004 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-serving-cert: secret "maas-api-e2e-extmod-a-c4fa79-serving-cert" not found Nov 03 04:09:37.041141 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.041021 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chjn\" (UniqueName: \"kubernetes.io/projected/ab3759f5-6d27-49c7-801d-11ca49d6d787-kube-api-access-8chjn\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.041141 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:09:37.041077 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls podName:ab3759f5-6d27-49c7-801d-11ca49d6d787 nodeName:}" failed. No retries permitted until 2026-11-03 04:09:37.541052458 +0000 UTC m=+4112.571650975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls") pod "maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" (UID: "ab3759f5-6d27-49c7-801d-11ca49d6d787") : secret "maas-api-e2e-extmod-a-c4fa79-serving-cert" not found Nov 03 04:09:37.041566 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.041544 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3759f5-6d27-49c7-801d-11ca49d6d787-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.043504 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.043484 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-metrics-certs\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.103781 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.103754 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chjn\" (UniqueName: \"kubernetes.io/projected/ab3759f5-6d27-49c7-801d-11ca49d6d787-kube-api-access-8chjn\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.548539 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.548503 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.551094 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.551068 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:37.779680 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:37.779644 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:38.170036 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:38.170011 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5"] Nov 03 04:09:38.170690 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:09:38.170665 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3759f5_6d27_49c7_801d_11ca49d6d787.slice/crio-ac95ea8d5fb4cb9a17cc6edec1b01e53b3361c9bcc954ecf623ca86111bb59d4 WatchSource:0}: Error finding container ac95ea8d5fb4cb9a17cc6edec1b01e53b3361c9bcc954ecf623ca86111bb59d4: Status 404 returned error can't find the container with id ac95ea8d5fb4cb9a17cc6edec1b01e53b3361c9bcc954ecf623ca86111bb59d4 Nov 03 04:09:38.437781 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:38.437699 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" event={"ID":"ab3759f5-6d27-49c7-801d-11ca49d6d787","Type":"ContainerStarted","Data":"ac95ea8d5fb4cb9a17cc6edec1b01e53b3361c9bcc954ecf623ca86111bb59d4"} Nov 03 04:09:39.447975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:39.447931 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" event={"ID":"ab3759f5-6d27-49c7-801d-11ca49d6d787","Type":"ContainerStarted","Data":"c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27"} Nov 03 04:09:39.448408 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:39.448067 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:09:39.567843 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:39.567787 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" podStartSLOduration=2.771501609 podStartE2EDuration="3.5677734s" totalImagesPullingTime="796.271791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:09:39.566912909 +0000 UTC m=+4114.597511448" watchObservedRunningTime="2026-11-03 04:09:39.5677734 +0000 UTC m=+4114.598372007" Nov 03 04:09:45.462194 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:09:45.462158 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:10:00.013694 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:10:00.013654 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:10:11.103343 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.103313 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj"] Nov 03 04:10:11.107527 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.107509 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.162790 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.162751 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-eed598-serving-cert\"" Nov 03 04:10:11.166916 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.166888 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj"] Nov 03 04:10:11.239284 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.239244 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-metrics-certs\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.239647 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.239620 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cad2cc0-f02a-4ced-b375-8f72f04291bd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.239867 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.239840 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.239971 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.239951 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzs7\" (UniqueName: \"kubernetes.io/projected/1cad2cc0-f02a-4ced-b375-8f72f04291bd-kube-api-access-9bzs7\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.340667 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.340628 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-metrics-certs\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.340855 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.340721 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cad2cc0-f02a-4ced-b375-8f72f04291bd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.340855 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.340745 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.340855 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.340776 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzs7\" (UniqueName: \"kubernetes.io/projected/1cad2cc0-f02a-4ced-b375-8f72f04291bd-kube-api-access-9bzs7\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.341035 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:10:11.340865 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-serving-cert: secret "maas-api-e2e-inf-a-eed598-serving-cert" not found Nov 03 04:10:11.341035 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:10:11.340933 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls podName:1cad2cc0-f02a-4ced-b375-8f72f04291bd nodeName:}" failed. No retries permitted until 2026-11-03 04:10:11.840910325 +0000 UTC m=+4146.871508841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls") pod "maas-api-e2e-inf-a-eed598-5d867f564c-876pj" (UID: "1cad2cc0-f02a-4ced-b375-8f72f04291bd") : secret "maas-api-e2e-inf-a-eed598-serving-cert" not found Nov 03 04:10:11.341638 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.341615 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cad2cc0-f02a-4ced-b375-8f72f04291bd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.343662 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.343642 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-metrics-certs\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.415653 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.415562 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzs7\" (UniqueName: \"kubernetes.io/projected/1cad2cc0-f02a-4ced-b375-8f72f04291bd-kube-api-access-9bzs7\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.846668 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.846633 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:11.849180 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:11.849154 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls\") pod \"maas-api-e2e-inf-a-eed598-5d867f564c-876pj\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:12.020817 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:12.020785 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:12.401435 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:12.401406 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj"] Nov 03 04:10:12.402286 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:10:12.402260 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cad2cc0_f02a_4ced_b375_8f72f04291bd.slice/crio-dfb1d6fda4a857ae1fac6ae6cc98197939b19eeac4c7a8ea573fe545cd90257d WatchSource:0}: Error finding container dfb1d6fda4a857ae1fac6ae6cc98197939b19eeac4c7a8ea573fe545cd90257d: Status 404 returned error can't find the container with id dfb1d6fda4a857ae1fac6ae6cc98197939b19eeac4c7a8ea573fe545cd90257d Nov 03 04:10:12.644241 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:12.644207 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" event={"ID":"1cad2cc0-f02a-4ced-b375-8f72f04291bd","Type":"ContainerStarted","Data":"dfb1d6fda4a857ae1fac6ae6cc98197939b19eeac4c7a8ea573fe545cd90257d"} Nov 03 04:10:13.651813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:13.651774 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" event={"ID":"1cad2cc0-f02a-4ced-b375-8f72f04291bd","Type":"ContainerStarted","Data":"7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c"} Nov 03 04:10:13.652290 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:13.651847 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:13.774285 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:13.774233 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" podStartSLOduration=1.976422755 podStartE2EDuration="2.77421783s" totalImagesPullingTime="797.795075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:10:13.771165947 +0000 UTC m=+4148.801764484" watchObservedRunningTime="2026-11-03 04:10:13.77421783 +0000 UTC m=+4148.804816368" Nov 03 04:10:19.664560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:10:19.664531 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:10:30.013051 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:10:30.013011 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:11:00.011632 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:11:00.011589 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:11:07.749095 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:07.749068 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:11:07.754012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:07.753989 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:11:07.762611 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:07.762582 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:11:07.768720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:07.768686 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:11:30.012292 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:11:30.012244 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:11:42.064427 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:42.064391 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn"] Nov 03 04:11:47.065123 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.065049 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" podUID="ab266756-b7f2-48b0-a26e-9be7c2deb5df" containerName="payload-processing" containerID="cri-o://444f2bdd82b0153519ed60d6e87bad13fe6ceba95e46b2b545d9a8f8327dc292" gracePeriod=25 Nov 03 04:11:47.262089 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.262057 5117 generic.go:356] "Generic (PLEG): container finished" podID="ab266756-b7f2-48b0-a26e-9be7c2deb5df" containerID="444f2bdd82b0153519ed60d6e87bad13fe6ceba95e46b2b545d9a8f8327dc292" exitCode=0 Nov 03 04:11:47.262253 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.262124 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" event={"ID":"ab266756-b7f2-48b0-a26e-9be7c2deb5df","Type":"ContainerDied","Data":"444f2bdd82b0153519ed60d6e87bad13fe6ceba95e46b2b545d9a8f8327dc292"} Nov 03 04:11:47.327557 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.327492 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:11:47.442507 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.442439 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab266756-b7f2-48b0-a26e-9be7c2deb5df-kube-api-access-cd4d2\" (UniqueName: \"kubernetes.io/projected/ab266756-b7f2-48b0-a26e-9be7c2deb5df-kube-api-access-cd4d2\") pod \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " Nov 03 04:11:47.442697 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.442558 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab266756-b7f2-48b0-a26e-9be7c2deb5df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab266756-b7f2-48b0-a26e-9be7c2deb5df-plugins-config-volume\") pod \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\" (UID: \"ab266756-b7f2-48b0-a26e-9be7c2deb5df\") " Nov 03 04:11:47.442963 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.442936 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab266756-b7f2-48b0-a26e-9be7c2deb5df-plugins-config-volume" pod "ab266756-b7f2-48b0-a26e-9be7c2deb5df" (UID: "ab266756-b7f2-48b0-a26e-9be7c2deb5df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:11:47.444864 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.444838 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab266756-b7f2-48b0-a26e-9be7c2deb5df-kube-api-access-cd4d2" pod "ab266756-b7f2-48b0-a26e-9be7c2deb5df" (UID: "ab266756-b7f2-48b0-a26e-9be7c2deb5df"). InnerVolumeSpecName "kube-api-access-cd4d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:11:47.544458 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.544410 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd4d2\" (UniqueName: \"kubernetes.io/projected/ab266756-b7f2-48b0-a26e-9be7c2deb5df-kube-api-access-cd4d2\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:47.544458 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:47.544439 5117 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab266756-b7f2-48b0-a26e-9be7c2deb5df-plugins-config-volume\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:11:48.270359 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:48.270329 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" Nov 03 04:11:48.270883 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:48.270354 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn" event={"ID":"ab266756-b7f2-48b0-a26e-9be7c2deb5df","Type":"ContainerDied","Data":"38ca00b1b77cdc3087e2777acccedd87a9757530303213cb75b44f20983047fb"} Nov 03 04:11:48.270883 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:48.270415 5117 scope.go:122] "RemoveContainer" containerID="444f2bdd82b0153519ed60d6e87bad13fe6ceba95e46b2b545d9a8f8327dc292" Nov 03 04:11:48.486995 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:48.486952 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn"] Nov 03 04:11:48.490196 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:48.490166 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-934bd4-cb64ddcb9-6h6rn"] Nov 03 04:11:49.090300 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:11:49.090263 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab266756-b7f2-48b0-a26e-9be7c2deb5df" path="/var/lib/kubelet/pods/ab266756-b7f2-48b0-a26e-9be7c2deb5df/volumes" Nov 03 04:12:00.015957 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:12:00.015913 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:12:30.012573 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:12:30.012531 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:12:33.103308 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.103269 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz"] Nov 03 04:12:33.103813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.103529 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" podUID="dc070a2e-08b9-4f80-8454-4a595fa55fa9" containerName="maas-api" containerID="cri-o://9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a" gracePeriod=30 Nov 03 04:12:33.405588 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.405555 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:12:33.563001 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.560569 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-metrics-certs\") pod \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " Nov 03 04:12:33.563001 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.560727 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc070a2e-08b9-4f80-8454-4a595fa55fa9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc070a2e-08b9-4f80-8454-4a595fa55fa9-openshift-service-ca\") pod \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " Nov 03 04:12:33.563001 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.560759 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc070a2e-08b9-4f80-8454-4a595fa55fa9-kube-api-access-5p62r\" (UniqueName: \"kubernetes.io/projected/dc070a2e-08b9-4f80-8454-4a595fa55fa9-kube-api-access-5p62r\") pod \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " Nov 03 04:12:33.563001 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.560799 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-maas-api-tls\") pod \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\" (UID: \"dc070a2e-08b9-4f80-8454-4a595fa55fa9\") " Nov 03 04:12:33.563001 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.562923 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc070a2e-08b9-4f80-8454-4a595fa55fa9-openshift-service-ca" pod "dc070a2e-08b9-4f80-8454-4a595fa55fa9" (UID: "dc070a2e-08b9-4f80-8454-4a595fa55fa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:12:33.564385 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.564336 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-maas-api-tls" pod "dc070a2e-08b9-4f80-8454-4a595fa55fa9" (UID: "dc070a2e-08b9-4f80-8454-4a595fa55fa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:12:33.564796 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.564668 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc070a2e-08b9-4f80-8454-4a595fa55fa9-kube-api-access-5p62r" pod "dc070a2e-08b9-4f80-8454-4a595fa55fa9" (UID: "dc070a2e-08b9-4f80-8454-4a595fa55fa9"). InnerVolumeSpecName "kube-api-access-5p62r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:12:33.567879 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.567853 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-metrics-certs" pod "dc070a2e-08b9-4f80-8454-4a595fa55fa9" (UID: "dc070a2e-08b9-4f80-8454-4a595fa55fa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:12:33.569540 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.569514 5117 generic.go:356] "Generic (PLEG): container finished" podID="dc070a2e-08b9-4f80-8454-4a595fa55fa9" containerID="9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a" exitCode=0 Nov 03 04:12:33.569666 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.569580 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" Nov 03 04:12:33.569666 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.569595 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" event={"ID":"dc070a2e-08b9-4f80-8454-4a595fa55fa9","Type":"ContainerDied","Data":"9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a"} Nov 03 04:12:33.569666 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.569648 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz" event={"ID":"dc070a2e-08b9-4f80-8454-4a595fa55fa9","Type":"ContainerDied","Data":"215b6d8354ec28020f8268f7f248e3e1b34024a91e078802286f937cc0784d34"} Nov 03 04:12:33.569822 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.569670 5117 scope.go:122] "RemoveContainer" containerID="9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a" Nov 03 04:12:33.598344 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.598315 5117 scope.go:122] "RemoveContainer" containerID="9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a" Nov 03 04:12:33.598786 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:12:33.598760 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a\": container with ID starting with 9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a not found: ID does not exist" containerID="9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a" Nov 03 04:12:33.598911 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.598795 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a"} err="failed to get container status \"9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a\": rpc error: code = NotFound desc = could not find container \"9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a\": container with ID starting with 9c5ca0d1b2f5d79cb7abf648c6c9cbcbabda729e8b3f4251e718f6cf9a258e8a not found: ID does not exist" Nov 03 04:12:33.662721 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.662637 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:33.662721 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.662669 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc070a2e-08b9-4f80-8454-4a595fa55fa9-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:33.662721 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.662682 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p62r\" (UniqueName: \"kubernetes.io/projected/dc070a2e-08b9-4f80-8454-4a595fa55fa9-kube-api-access-5p62r\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:33.662721 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.662691 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc070a2e-08b9-4f80-8454-4a595fa55fa9-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:33.739498 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.739455 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz"] Nov 03 04:12:33.743243 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:33.743210 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-214eab-5f4b565f86-ptxlz"] Nov 03 04:12:35.093818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:35.093755 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc070a2e-08b9-4f80-8454-4a595fa55fa9" path="/var/lib/kubelet/pods/dc070a2e-08b9-4f80-8454-4a595fa55fa9/volumes" Nov 03 04:12:39.770848 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.770801 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm"] Nov 03 04:12:39.771693 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.771668 5117 cpu_manager.go:414] "containerMap: removing container" podUID="ab266756-b7f2-48b0-a26e-9be7c2deb5df" containerName="payload-processing" Nov 03 04:12:39.771814 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.771694 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab266756-b7f2-48b0-a26e-9be7c2deb5df" containerName="payload-processing" Nov 03 04:12:39.771814 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.771739 5117 cpu_manager.go:414] "containerMap: removing container" podUID="dc070a2e-08b9-4f80-8454-4a595fa55fa9" containerName="maas-api" Nov 03 04:12:39.771814 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.771748 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc070a2e-08b9-4f80-8454-4a595fa55fa9" containerName="maas-api" Nov 03 04:12:39.771975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.771928 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab266756-b7f2-48b0-a26e-9be7c2deb5df" containerName="payload-processing" Nov 03 04:12:39.771975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.771950 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc070a2e-08b9-4f80-8454-4a595fa55fa9" containerName="maas-api" Nov 03 04:12:39.775598 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.775571 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.830376 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.830334 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr5nn\" (UniqueName: \"kubernetes.io/projected/da0aaa21-4191-4fd5-8624-b9e75068a9e0-kube-api-access-xr5nn\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.830605 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.830401 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.830605 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.830526 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-metrics-certs\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.830705 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.830636 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0aaa21-4191-4fd5-8624-b9e75068a9e0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.880997 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.880961 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8540f6-serving-cert\"" Nov 03 04:12:39.884116 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.884083 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm"] Nov 03 04:12:39.932211 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.932177 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.932432 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.932232 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-metrics-certs\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.932432 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:12:39.932354 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-serving-cert: secret "maas-api-e2e-inf-b-8540f6-serving-cert" not found Nov 03 04:12:39.932432 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.932358 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0aaa21-4191-4fd5-8624-b9e75068a9e0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.932432 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:12:39.932420 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls podName:da0aaa21-4191-4fd5-8624-b9e75068a9e0 nodeName:}" failed. No retries permitted until 2026-11-03 04:12:40.432403562 +0000 UTC m=+4295.463002108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls") pod "maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" (UID: "da0aaa21-4191-4fd5-8624-b9e75068a9e0") : secret "maas-api-e2e-inf-b-8540f6-serving-cert" not found Nov 03 04:12:39.932679 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.932569 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr5nn\" (UniqueName: \"kubernetes.io/projected/da0aaa21-4191-4fd5-8624-b9e75068a9e0-kube-api-access-xr5nn\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.933104 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.933085 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0aaa21-4191-4fd5-8624-b9e75068a9e0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.935059 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.935040 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-metrics-certs\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:39.995239 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:39.995205 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr5nn\" (UniqueName: \"kubernetes.io/projected/da0aaa21-4191-4fd5-8624-b9e75068a9e0-kube-api-access-xr5nn\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:40.439009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:40.438965 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:40.441839 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:40.441809 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:40.689129 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:40.689029 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:41.074953 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:41.074918 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm"] Nov 03 04:12:41.075842 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:12:41.075813 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0aaa21_4191_4fd5_8624_b9e75068a9e0.slice/crio-fcb10f4b3aa22780667c025341cbb9f3fe1ee3f8fd5aeecc3f250733bcf9c04f WatchSource:0}: Error finding container fcb10f4b3aa22780667c025341cbb9f3fe1ee3f8fd5aeecc3f250733bcf9c04f: Status 404 returned error can't find the container with id fcb10f4b3aa22780667c025341cbb9f3fe1ee3f8fd5aeecc3f250733bcf9c04f Nov 03 04:12:41.632534 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:41.632486 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" event={"ID":"da0aaa21-4191-4fd5-8624-b9e75068a9e0","Type":"ContainerStarted","Data":"fcb10f4b3aa22780667c025341cbb9f3fe1ee3f8fd5aeecc3f250733bcf9c04f"} Nov 03 04:12:42.641788 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:42.641747 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" event={"ID":"da0aaa21-4191-4fd5-8624-b9e75068a9e0","Type":"ContainerStarted","Data":"35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd"} Nov 03 04:12:42.642222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:42.641906 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:12:42.765583 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:42.765513 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" podStartSLOduration=2.834269374 podStartE2EDuration="3.76549174s" totalImagesPullingTime="931.222366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:12:42.761685112 +0000 UTC m=+4297.792283653" watchObservedRunningTime="2026-11-03 04:12:42.76549174 +0000 UTC m=+4297.796090281" Nov 03 04:12:44.640692 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.640611 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5"] Nov 03 04:12:44.641233 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.640848 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" podUID="ab3759f5-6d27-49c7-801d-11ca49d6d787" containerName="maas-api" containerID="cri-o://c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27" gracePeriod=30 Nov 03 04:12:44.936770 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.936742 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:12:44.992802 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.992761 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3759f5-6d27-49c7-801d-11ca49d6d787-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3759f5-6d27-49c7-801d-11ca49d6d787-openshift-service-ca\") pod \"ab3759f5-6d27-49c7-801d-11ca49d6d787\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " Nov 03 04:12:44.993008 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.992839 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3759f5-6d27-49c7-801d-11ca49d6d787-kube-api-access-8chjn\" (UniqueName: \"kubernetes.io/projected/ab3759f5-6d27-49c7-801d-11ca49d6d787-kube-api-access-8chjn\") pod \"ab3759f5-6d27-49c7-801d-11ca49d6d787\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " Nov 03 04:12:44.993008 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.992900 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls\") pod \"ab3759f5-6d27-49c7-801d-11ca49d6d787\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " Nov 03 04:12:44.993124 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.993047 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-metrics-certs\") pod \"ab3759f5-6d27-49c7-801d-11ca49d6d787\" (UID: \"ab3759f5-6d27-49c7-801d-11ca49d6d787\") " Nov 03 04:12:44.993252 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.993222 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3759f5-6d27-49c7-801d-11ca49d6d787-openshift-service-ca" pod "ab3759f5-6d27-49c7-801d-11ca49d6d787" (UID: "ab3759f5-6d27-49c7-801d-11ca49d6d787"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:12:44.993561 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.993535 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3759f5-6d27-49c7-801d-11ca49d6d787-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:44.995318 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.995284 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-metrics-certs" pod "ab3759f5-6d27-49c7-801d-11ca49d6d787" (UID: "ab3759f5-6d27-49c7-801d-11ca49d6d787"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:12:44.995468 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.995362 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3759f5-6d27-49c7-801d-11ca49d6d787-kube-api-access-8chjn" pod "ab3759f5-6d27-49c7-801d-11ca49d6d787" (UID: "ab3759f5-6d27-49c7-801d-11ca49d6d787"). InnerVolumeSpecName "kube-api-access-8chjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:12:44.995629 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:44.995607 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls" pod "ab3759f5-6d27-49c7-801d-11ca49d6d787" (UID: "ab3759f5-6d27-49c7-801d-11ca49d6d787"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:12:45.094605 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.094576 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:45.094802 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.094784 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chjn\" (UniqueName: \"kubernetes.io/projected/ab3759f5-6d27-49c7-801d-11ca49d6d787-kube-api-access-8chjn\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:45.094880 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.094805 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3759f5-6d27-49c7-801d-11ca49d6d787-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:12:45.665459 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.665385 5117 generic.go:356] "Generic (PLEG): container finished" podID="ab3759f5-6d27-49c7-801d-11ca49d6d787" containerID="c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27" exitCode=0 Nov 03 04:12:45.666251 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.666217 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" event={"ID":"ab3759f5-6d27-49c7-801d-11ca49d6d787","Type":"ContainerDied","Data":"c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27"} Nov 03 04:12:45.666371 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.666287 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" event={"ID":"ab3759f5-6d27-49c7-801d-11ca49d6d787","Type":"ContainerDied","Data":"ac95ea8d5fb4cb9a17cc6edec1b01e53b3361c9bcc954ecf623ca86111bb59d4"} Nov 03 04:12:45.666371 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.666322 5117 scope.go:122] "RemoveContainer" containerID="c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27" Nov 03 04:12:45.666547 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.666529 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5" Nov 03 04:12:45.681015 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.680984 5117 scope.go:122] "RemoveContainer" containerID="c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27" Nov 03 04:12:45.681483 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:12:45.681411 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27\": container with ID starting with c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27 not found: ID does not exist" containerID="c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27" Nov 03 04:12:45.681608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.681530 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27"} err="failed to get container status \"c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27\": rpc error: code = NotFound desc = could not find container \"c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27\": container with ID starting with c025e7d3884010241b9e5ed2af2da2a52847f48aeba8401eec5ea60aee309b27 not found: ID does not exist" Nov 03 04:12:45.847513 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.847431 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5"] Nov 03 04:12:45.898593 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:45.898557 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4fa79-c4f977d8c-7n5m5"] Nov 03 04:12:47.092658 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:47.092623 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3759f5-6d27-49c7-801d-11ca49d6d787" path="/var/lib/kubelet/pods/ab3759f5-6d27-49c7-801d-11ca49d6d787/volumes" Nov 03 04:12:48.653517 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:12:48.653483 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:13:00.013136 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:13:00.013091 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:13:14.025599 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:13:14.025553 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:13:14.043469 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:13:14.026325 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:13:30.012064 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:13:30.012020 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:14:00.013053 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:14:00.013003 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:14:18.031073 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.031038 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2"] Nov 03 04:14:18.031693 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.031677 5117 cpu_manager.go:414] "containerMap: removing container" podUID="ab3759f5-6d27-49c7-801d-11ca49d6d787" containerName="maas-api" Nov 03 04:14:18.031758 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.031694 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3759f5-6d27-49c7-801d-11ca49d6d787" containerName="maas-api" Nov 03 04:14:18.031800 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.031791 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3759f5-6d27-49c7-801d-11ca49d6d787" containerName="maas-api" Nov 03 04:14:18.035258 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.035237 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.090278 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.090241 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-255c49-serving-cert\"" Nov 03 04:14:18.143868 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.143821 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.144054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.143950 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-metrics-certs\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.144054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.144024 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8070701-d6c6-4399-88d0-298b5fca3efb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.144054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.144047 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ngf\" (UniqueName: \"kubernetes.io/projected/c8070701-d6c6-4399-88d0-298b5fca3efb-kube-api-access-q8ngf\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.144648 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.144627 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2"] Nov 03 04:14:18.245381 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.245335 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8070701-d6c6-4399-88d0-298b5fca3efb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.245608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.245390 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ngf\" (UniqueName: \"kubernetes.io/projected/c8070701-d6c6-4399-88d0-298b5fca3efb-kube-api-access-q8ngf\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.245608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.245578 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.245739 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.245639 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-metrics-certs\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.245739 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:14:18.245719 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-serving-cert: secret "maas-api-e2e-inf-a-255c49-serving-cert" not found Nov 03 04:14:18.245859 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:14:18.245797 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls podName:c8070701-d6c6-4399-88d0-298b5fca3efb nodeName:}" failed. No retries permitted until 2026-11-03 04:14:18.745770684 +0000 UTC m=+4393.776369215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls") pod "maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" (UID: "c8070701-d6c6-4399-88d0-298b5fca3efb") : secret "maas-api-e2e-inf-a-255c49-serving-cert" not found Nov 03 04:14:18.246099 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.246075 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8070701-d6c6-4399-88d0-298b5fca3efb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.248352 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.248323 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-metrics-certs\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.309242 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.309199 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ngf\" (UniqueName: \"kubernetes.io/projected/c8070701-d6c6-4399-88d0-298b5fca3efb-kube-api-access-q8ngf\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.751711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.751622 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.754560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.754532 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls\") pod \"maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:18.949347 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:18.949300 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:19.330044 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:19.329998 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2"] Nov 03 04:14:19.330537 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:14:19.330497 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8070701_d6c6_4399_88d0_298b5fca3efb.slice/crio-314d517672791b9ba4ba96d6dc8b4623d696cce8973a411c0b02f1b3b2548510 WatchSource:0}: Error finding container 314d517672791b9ba4ba96d6dc8b4623d696cce8973a411c0b02f1b3b2548510: Status 404 returned error can't find the container with id 314d517672791b9ba4ba96d6dc8b4623d696cce8973a411c0b02f1b3b2548510 Nov 03 04:14:19.332078 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:19.332060 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 04:14:20.316820 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:20.316784 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" event={"ID":"c8070701-d6c6-4399-88d0-298b5fca3efb","Type":"ContainerStarted","Data":"d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6"} Nov 03 04:14:20.317037 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:20.316848 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" event={"ID":"c8070701-d6c6-4399-88d0-298b5fca3efb","Type":"ContainerStarted","Data":"314d517672791b9ba4ba96d6dc8b4623d696cce8973a411c0b02f1b3b2548510"} Nov 03 04:14:20.317037 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:20.316947 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:20.453256 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:20.453149 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" podStartSLOduration=2.6318610810000003 podStartE2EDuration="3.453134157s" totalImagesPullingTime="821.273076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:14:20.451699733 +0000 UTC m=+4395.482298272" watchObservedRunningTime="2026-11-03 04:14:20.453134157 +0000 UTC m=+4395.483732694" Nov 03 04:14:26.329086 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:26.329056 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:14:30.011493 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:14:30.011433 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:14:38.882226 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:38.882187 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj"] Nov 03 04:14:38.882839 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:38.882389 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" podUID="1cad2cc0-f02a-4ced-b375-8f72f04291bd" containerName="maas-api" containerID="cri-o://7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c" gracePeriod=30 Nov 03 04:14:39.187286 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.187250 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:14:39.287243 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.287199 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cad2cc0-f02a-4ced-b375-8f72f04291bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cad2cc0-f02a-4ced-b375-8f72f04291bd-openshift-service-ca\") pod \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " Nov 03 04:14:39.287488 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.287258 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls\") pod \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " Nov 03 04:14:39.287488 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.287302 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-metrics-certs\") pod \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " Nov 03 04:14:39.287488 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.287352 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cad2cc0-f02a-4ced-b375-8f72f04291bd-kube-api-access-9bzs7\" (UniqueName: \"kubernetes.io/projected/1cad2cc0-f02a-4ced-b375-8f72f04291bd-kube-api-access-9bzs7\") pod \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\" (UID: \"1cad2cc0-f02a-4ced-b375-8f72f04291bd\") " Nov 03 04:14:39.291224 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.291187 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls" pod "1cad2cc0-f02a-4ced-b375-8f72f04291bd" (UID: "1cad2cc0-f02a-4ced-b375-8f72f04291bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:14:39.291408 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.291249 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cad2cc0-f02a-4ced-b375-8f72f04291bd-openshift-service-ca" pod "1cad2cc0-f02a-4ced-b375-8f72f04291bd" (UID: "1cad2cc0-f02a-4ced-b375-8f72f04291bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:14:39.294501 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.293712 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-metrics-certs" pod "1cad2cc0-f02a-4ced-b375-8f72f04291bd" (UID: "1cad2cc0-f02a-4ced-b375-8f72f04291bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:14:39.296086 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.296053 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cad2cc0-f02a-4ced-b375-8f72f04291bd-kube-api-access-9bzs7" pod "1cad2cc0-f02a-4ced-b375-8f72f04291bd" (UID: "1cad2cc0-f02a-4ced-b375-8f72f04291bd"). InnerVolumeSpecName "kube-api-access-9bzs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:14:39.389283 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.389249 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cad2cc0-f02a-4ced-b375-8f72f04291bd-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:39.389283 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.389281 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:39.389496 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.389292 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cad2cc0-f02a-4ced-b375-8f72f04291bd-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:39.389496 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.389301 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzs7\" (UniqueName: \"kubernetes.io/projected/1cad2cc0-f02a-4ced-b375-8f72f04291bd-kube-api-access-9bzs7\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:14:39.449243 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.449145 5117 generic.go:356] "Generic (PLEG): container finished" podID="1cad2cc0-f02a-4ced-b375-8f72f04291bd" containerID="7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c" exitCode=0 Nov 03 04:14:39.449537 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.449276 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" event={"ID":"1cad2cc0-f02a-4ced-b375-8f72f04291bd","Type":"ContainerDied","Data":"7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c"} Nov 03 04:14:39.449537 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.449382 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" event={"ID":"1cad2cc0-f02a-4ced-b375-8f72f04291bd","Type":"ContainerDied","Data":"dfb1d6fda4a857ae1fac6ae6cc98197939b19eeac4c7a8ea573fe545cd90257d"} Nov 03 04:14:39.449537 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.449428 5117 scope.go:122] "RemoveContainer" containerID="7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c" Nov 03 04:14:39.449719 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.449638 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj" Nov 03 04:14:39.465396 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.465375 5117 scope.go:122] "RemoveContainer" containerID="7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c" Nov 03 04:14:39.465712 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:14:39.465694 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c\": container with ID starting with 7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c not found: ID does not exist" containerID="7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c" Nov 03 04:14:39.465790 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.465727 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c"} err="failed to get container status \"7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c\": rpc error: code = NotFound desc = could not find container \"7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c\": container with ID starting with 7d89e267359ec64aa0acd4593204e0e7c312b78878326b70a089a5202211728c not found: ID does not exist" Nov 03 04:14:39.621613 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.621578 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj"] Nov 03 04:14:39.624374 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:39.624350 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eed598-5d867f564c-876pj"] Nov 03 04:14:41.090240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:14:41.090208 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cad2cc0-f02a-4ced-b375-8f72f04291bd" path="/var/lib/kubelet/pods/1cad2cc0-f02a-4ced-b375-8f72f04291bd/volumes" Nov 03 04:15:00.013589 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:15:00.013547 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:15:20.238963 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.238869 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj"] Nov 03 04:15:20.239564 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.239118 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" podUID="8d8b4818-a5e0-4b3f-b82f-479d243dd14c" containerName="maas-api" containerID="cri-o://3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715" gracePeriod=30 Nov 03 04:15:20.567631 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.567599 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:15:20.684768 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.684717 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls\") pod \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " Nov 03 04:15:20.684989 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.684969 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-openshift-service-ca\") pod \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " Nov 03 04:15:20.685075 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.685013 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-kube-api-access-dljkc\" (UniqueName: \"kubernetes.io/projected/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-kube-api-access-dljkc\") pod \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " Nov 03 04:15:20.685075 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.685049 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-metrics-certs\") pod \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\" (UID: \"8d8b4818-a5e0-4b3f-b82f-479d243dd14c\") " Nov 03 04:15:20.686421 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.686390 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-openshift-service-ca" pod "8d8b4818-a5e0-4b3f-b82f-479d243dd14c" (UID: "8d8b4818-a5e0-4b3f-b82f-479d243dd14c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:15:20.689530 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.689486 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-metrics-certs" pod "8d8b4818-a5e0-4b3f-b82f-479d243dd14c" (UID: "8d8b4818-a5e0-4b3f-b82f-479d243dd14c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:15:20.689669 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.689551 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls" pod "8d8b4818-a5e0-4b3f-b82f-479d243dd14c" (UID: "8d8b4818-a5e0-4b3f-b82f-479d243dd14c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:15:20.692565 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.692511 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-kube-api-access-dljkc" pod "8d8b4818-a5e0-4b3f-b82f-479d243dd14c" (UID: "8d8b4818-a5e0-4b3f-b82f-479d243dd14c"). InnerVolumeSpecName "kube-api-access-dljkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:15:20.719754 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.719723 5117 generic.go:356] "Generic (PLEG): container finished" podID="8d8b4818-a5e0-4b3f-b82f-479d243dd14c" containerID="3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715" exitCode=0 Nov 03 04:15:20.719919 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.719769 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" Nov 03 04:15:20.719919 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.719787 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" event={"ID":"8d8b4818-a5e0-4b3f-b82f-479d243dd14c","Type":"ContainerDied","Data":"3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715"} Nov 03 04:15:20.719919 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.719832 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj" event={"ID":"8d8b4818-a5e0-4b3f-b82f-479d243dd14c","Type":"ContainerDied","Data":"02f5663f69a74e3d613b9a5e928284627107379ed78d415b86601c3a15de1917"} Nov 03 04:15:20.719919 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.719855 5117 scope.go:122] "RemoveContainer" containerID="3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715" Nov 03 04:15:20.730595 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.730572 5117 scope.go:122] "RemoveContainer" containerID="3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715" Nov 03 04:15:20.730910 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:15:20.730883 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715\": container with ID starting with 3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715 not found: ID does not exist" containerID="3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715" Nov 03 04:15:20.730996 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.730922 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715"} err="failed to get container status \"3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715\": rpc error: code = NotFound desc = could not find container \"3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715\": container with ID starting with 3cb497e5127e2a0e6a0665b3a01d94c9034a65e2a61b5a807b0ab0fe02097715 not found: ID does not exist" Nov 03 04:15:20.787079 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.787051 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.787079 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.787076 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.787258 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.787086 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dljkc\" (UniqueName: \"kubernetes.io/projected/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-kube-api-access-dljkc\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.787258 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.787095 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8b4818-a5e0-4b3f-b82f-479d243dd14c-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:15:20.887299 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.887263 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj"] Nov 03 04:15:20.895655 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:20.895624 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed4ecc-6bf6fbb967-qfdkj"] Nov 03 04:15:21.089968 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:15:21.089926 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8b4818-a5e0-4b3f-b82f-479d243dd14c" path="/var/lib/kubelet/pods/8d8b4818-a5e0-4b3f-b82f-479d243dd14c/volumes" Nov 03 04:15:30.011385 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:15:30.011347 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:16:00.016567 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:16:00.016512 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:16:07.792160 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:07.792001 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:16:07.798515 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:07.797146 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:16:07.817632 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:07.817608 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:16:07.822936 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:07.822917 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:16:12.789568 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.789531 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk"] Nov 03 04:16:12.790094 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.790080 5117 cpu_manager.go:414] "containerMap: removing container" podUID="1cad2cc0-f02a-4ced-b375-8f72f04291bd" containerName="maas-api" Nov 03 04:16:12.790094 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.790094 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cad2cc0-f02a-4ced-b375-8f72f04291bd" containerName="maas-api" Nov 03 04:16:12.790174 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.790125 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8d8b4818-a5e0-4b3f-b82f-479d243dd14c" containerName="maas-api" Nov 03 04:16:12.790174 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.790131 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8b4818-a5e0-4b3f-b82f-479d243dd14c" containerName="maas-api" Nov 03 04:16:12.790242 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.790203 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8b4818-a5e0-4b3f-b82f-479d243dd14c" containerName="maas-api" Nov 03 04:16:12.790242 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.790215 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cad2cc0-f02a-4ced-b375-8f72f04291bd" containerName="maas-api" Nov 03 04:16:12.793155 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.793138 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.848085 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.848055 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e30496c2-serving-cert\"" Nov 03 04:16:12.860740 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.860712 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlz8q\" (UniqueName: \"kubernetes.io/projected/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-kube-api-access-mlz8q\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.860896 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.860761 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-metrics-certs\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.860968 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.860900 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.860968 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.860938 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-openshift-service-ca\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.901683 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.901658 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk"] Nov 03 04:16:12.962429 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.962394 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-openshift-service-ca\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.962818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.962786 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlz8q\" (UniqueName: \"kubernetes.io/projected/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-kube-api-access-mlz8q\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.962933 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.962859 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-metrics-certs\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.962999 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.962964 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.963103 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:16:12.963087 5117 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-serving-cert: secret "maas-api-e2e-mt-e30496c2-serving-cert" not found Nov 03 04:16:12.963161 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:16:12.963153 5117 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls podName:d4d334d3-ed81-4de0-9422-b9f8e1d123ad nodeName:}" failed. No retries permitted until 2026-11-03 04:16:13.463132068 +0000 UTC m=+4508.493730598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls") pod "maas-api-e2e-mt-e30496c2-5d79448777-p8srk" (UID: "d4d334d3-ed81-4de0-9422-b9f8e1d123ad") : secret "maas-api-e2e-mt-e30496c2-serving-cert" not found Nov 03 04:16:12.963533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.963510 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-openshift-service-ca\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:12.965639 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:12.965620 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-metrics-certs\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:13.025432 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:13.025404 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlz8q\" (UniqueName: \"kubernetes.io/projected/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-kube-api-access-mlz8q\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:13.469269 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:13.469235 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:13.471875 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:13.471848 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls\") pod \"maas-api-e2e-mt-e30496c2-5d79448777-p8srk\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:13.706643 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:13.706612 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:14.087757 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:14.087727 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk"] Nov 03 04:16:14.093557 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:16:14.093521 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d334d3_ed81_4de0_9422_b9f8e1d123ad.slice/crio-fffb7f3c156254dd3f6a770cef31e6db13b75d6642d2fca33175c3cd5738d9f8 WatchSource:0}: Error finding container fffb7f3c156254dd3f6a770cef31e6db13b75d6642d2fca33175c3cd5738d9f8: Status 404 returned error can't find the container with id fffb7f3c156254dd3f6a770cef31e6db13b75d6642d2fca33175c3cd5738d9f8 Nov 03 04:16:15.058918 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:15.058868 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" event={"ID":"d4d334d3-ed81-4de0-9422-b9f8e1d123ad","Type":"ContainerStarted","Data":"fffb7f3c156254dd3f6a770cef31e6db13b75d6642d2fca33175c3cd5738d9f8"} Nov 03 04:16:16.065892 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:16.065841 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" event={"ID":"d4d334d3-ed81-4de0-9422-b9f8e1d123ad","Type":"ContainerStarted","Data":"c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4"} Nov 03 04:16:16.066581 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:16.066518 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:16.186599 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:16.186546 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" podStartSLOduration=3.279447012 podStartE2EDuration="4.186532134s" totalImagesPullingTime="907.085122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:16:16.18501621 +0000 UTC m=+4511.215614753" watchObservedRunningTime="2026-11-03 04:16:16.186532134 +0000 UTC m=+4511.217130672" Nov 03 04:16:23.091691 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:23.091649 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:16:30.011030 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:16:30.010989 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:16:51.356924 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.356839 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm"] Nov 03 04:16:51.357520 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.357051 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" podUID="da0aaa21-4191-4fd5-8624-b9e75068a9e0" containerName="maas-api" containerID="cri-o://35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd" gracePeriod=30 Nov 03 04:16:51.646705 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.646674 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:16:51.691744 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.691715 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-metrics-certs\") pod \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " Nov 03 04:16:51.691899 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.691883 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0aaa21-4191-4fd5-8624-b9e75068a9e0-kube-api-access-xr5nn\" (UniqueName: \"kubernetes.io/projected/da0aaa21-4191-4fd5-8624-b9e75068a9e0-kube-api-access-xr5nn\") pod \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " Nov 03 04:16:51.691962 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.691947 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls\") pod \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " Nov 03 04:16:51.692008 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.691984 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0aaa21-4191-4fd5-8624-b9e75068a9e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0aaa21-4191-4fd5-8624-b9e75068a9e0-openshift-service-ca\") pod \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\" (UID: \"da0aaa21-4191-4fd5-8624-b9e75068a9e0\") " Nov 03 04:16:51.692524 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.692479 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0aaa21-4191-4fd5-8624-b9e75068a9e0-openshift-service-ca" pod "da0aaa21-4191-4fd5-8624-b9e75068a9e0" (UID: "da0aaa21-4191-4fd5-8624-b9e75068a9e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:16:51.694406 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.694380 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-metrics-certs" pod "da0aaa21-4191-4fd5-8624-b9e75068a9e0" (UID: "da0aaa21-4191-4fd5-8624-b9e75068a9e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:16:51.694776 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.694751 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls" pod "da0aaa21-4191-4fd5-8624-b9e75068a9e0" (UID: "da0aaa21-4191-4fd5-8624-b9e75068a9e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:16:51.694829 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.694800 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0aaa21-4191-4fd5-8624-b9e75068a9e0-kube-api-access-xr5nn" pod "da0aaa21-4191-4fd5-8624-b9e75068a9e0" (UID: "da0aaa21-4191-4fd5-8624-b9e75068a9e0"). InnerVolumeSpecName "kube-api-access-xr5nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:16:51.793319 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.793285 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:16:51.793319 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.793319 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0aaa21-4191-4fd5-8624-b9e75068a9e0-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:16:51.793581 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.793333 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0aaa21-4191-4fd5-8624-b9e75068a9e0-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:16:51.793581 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:51.793345 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr5nn\" (UniqueName: \"kubernetes.io/projected/da0aaa21-4191-4fd5-8624-b9e75068a9e0-kube-api-access-xr5nn\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:16:52.295459 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.295404 5117 generic.go:356] "Generic (PLEG): container finished" podID="da0aaa21-4191-4fd5-8624-b9e75068a9e0" containerID="35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd" exitCode=0 Nov 03 04:16:52.295657 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.295475 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" Nov 03 04:16:52.295657 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.295485 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" event={"ID":"da0aaa21-4191-4fd5-8624-b9e75068a9e0","Type":"ContainerDied","Data":"35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd"} Nov 03 04:16:52.295657 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.295542 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm" event={"ID":"da0aaa21-4191-4fd5-8624-b9e75068a9e0","Type":"ContainerDied","Data":"fcb10f4b3aa22780667c025341cbb9f3fe1ee3f8fd5aeecc3f250733bcf9c04f"} Nov 03 04:16:52.295657 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.295566 5117 scope.go:122] "RemoveContainer" containerID="35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd" Nov 03 04:16:52.306914 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.306885 5117 scope.go:122] "RemoveContainer" containerID="35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd" Nov 03 04:16:52.307262 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:16:52.307238 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd\": container with ID starting with 35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd not found: ID does not exist" containerID="35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd" Nov 03 04:16:52.307346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.307272 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd"} err="failed to get container status \"35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd\": rpc error: code = NotFound desc = could not find container \"35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd\": container with ID starting with 35fc3c690c94dade9b6028ca6a4f9d72076957edd0605442503def04cb0dc0dd not found: ID does not exist" Nov 03 04:16:52.463850 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.463814 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm"] Nov 03 04:16:52.514710 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:52.514678 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8540f6-7b87c4cf8c-qztkm"] Nov 03 04:16:53.091267 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:16:53.091235 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0aaa21-4191-4fd5-8624-b9e75068a9e0" path="/var/lib/kubelet/pods/da0aaa21-4191-4fd5-8624-b9e75068a9e0/volumes" Nov 03 04:17:00.012840 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:17:00.012801 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:17:02.528821 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.528787 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf"] Nov 03 04:17:02.529377 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.529354 5117 cpu_manager.go:414] "containerMap: removing container" podUID="da0aaa21-4191-4fd5-8624-b9e75068a9e0" containerName="maas-api" Nov 03 04:17:02.529377 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.529370 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0aaa21-4191-4fd5-8624-b9e75068a9e0" containerName="maas-api" Nov 03 04:17:02.529596 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.529493 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0aaa21-4191-4fd5-8624-b9e75068a9e0" containerName="maas-api" Nov 03 04:17:02.534071 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.534050 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.590955 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.590923 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf"] Nov 03 04:17:02.638412 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.638376 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"kube-root-ca.crt\"" Nov 03 04:17:02.638644 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.638440 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"openshift-service-ca.crt\"" Nov 03 04:17:02.638644 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.638527 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"default-dockercfg-c5z2n\"" Nov 03 04:17:02.638644 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.638586 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-66d902\"/\"rate-test-model-66d902-kserve-self-signed-certs\"" Nov 03 04:17:02.638877 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.638859 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-dshm\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.638917 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.638896 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-model-cache\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.638962 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.638929 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzqp\" (UniqueName: \"kubernetes.io/projected/3f3f800d-e685-42ae-b7a8-df2ede6cb623-kube-api-access-7dzqp\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.639078 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.639058 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tmp-dir\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.639171 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.639155 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tls-certs\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.639225 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.639197 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-home\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.740807 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.740762 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-dshm\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.740831 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-model-cache\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.740883 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzqp\" (UniqueName: \"kubernetes.io/projected/3f3f800d-e685-42ae-b7a8-df2ede6cb623-kube-api-access-7dzqp\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.740922 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tmp-dir\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.740989 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tls-certs\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741238 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.741038 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-home\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741356 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.741330 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-model-cache\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741433 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.741392 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tmp-dir\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.741524 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.741433 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-home\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.743489 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.743434 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-dshm\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.744316 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.744297 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tls-certs\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.804259 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.804183 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzqp\" (UniqueName: \"kubernetes.io/projected/3f3f800d-e685-42ae-b7a8-df2ede6cb623-kube-api-access-7dzqp\") pod \"rate-test-model-66d902-kserve-f6f9fd9df-876pf\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:02.846569 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:02.846528 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:03.236962 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:03.236932 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf"] Nov 03 04:17:03.237898 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:17:03.237870 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f3f800d_e685_42ae_b7a8_df2ede6cb623.slice/crio-01cd09d17209e54a88dc5b99ae37b1a2726f96e7de43da7b1faa5937b2f2fbd3 WatchSource:0}: Error finding container 01cd09d17209e54a88dc5b99ae37b1a2726f96e7de43da7b1faa5937b2f2fbd3: Status 404 returned error can't find the container with id 01cd09d17209e54a88dc5b99ae37b1a2726f96e7de43da7b1faa5937b2f2fbd3 Nov 03 04:17:03.425834 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:03.425797 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" event={"ID":"3f3f800d-e685-42ae-b7a8-df2ede6cb623","Type":"ContainerStarted","Data":"316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07"} Nov 03 04:17:03.426002 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:03.425849 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" event={"ID":"3f3f800d-e685-42ae-b7a8-df2ede6cb623","Type":"ContainerStarted","Data":"01cd09d17209e54a88dc5b99ae37b1a2726f96e7de43da7b1faa5937b2f2fbd3"} Nov 03 04:17:03.569735 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:03.569670 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" podStartSLOduration=1.569650958 podStartE2EDuration="1.569650958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:17:03.546232409 +0000 UTC m=+4558.576830948" watchObservedRunningTime="2026-11-03 04:17:03.569650958 +0000 UTC m=+4558.600249498" Nov 03 04:17:03.847404 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:03.847324 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:12.846734 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:12.846694 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:12.859030 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:12.859001 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:13.508227 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:13.508199 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:17:30.012103 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:17:30.012060 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:17:41.618240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:41.618197 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq"] Nov 03 04:17:56.640297 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.640239 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" containerID="cri-o://d7e2c6c780a7f4251a421fbac5eba26aa70bbd12068e031ae0fb81bbd519b948" gracePeriod=45 Nov 03 04:17:56.771039 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.770999 5117 generic.go:356] "Generic (PLEG): container finished" podID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerID="d7e2c6c780a7f4251a421fbac5eba26aa70bbd12068e031ae0fb81bbd519b948" exitCode=0 Nov 03 04:17:56.771162 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.771073 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" event={"ID":"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f","Type":"ContainerDied","Data":"d7e2c6c780a7f4251a421fbac5eba26aa70bbd12068e031ae0fb81bbd519b948"} Nov 03 04:17:56.909102 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.909042 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 04:17:56.987376 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987348 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-model-cache\") pod \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " Nov 03 04:17:56.987578 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987383 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-kube-api-access-nx6z8\" (UniqueName: \"kubernetes.io/projected/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-kube-api-access-nx6z8\") pod \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " Nov 03 04:17:56.987578 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987426 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-home\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-home\") pod \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " Nov 03 04:17:56.987578 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987527 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tmp-dir\") pod \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " Nov 03 04:17:56.987578 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987560 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-dshm\") pod \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " Nov 03 04:17:56.987818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987648 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-model-cache" pod "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" (UID: "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:56.987818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987668 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tls-certs\") pod \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\" (UID: \"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f\") " Nov 03 04:17:56.987818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987689 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-home" pod "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" (UID: "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:56.987818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.987768 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tmp-dir" pod "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" (UID: "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:56.988485 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.988429 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:56.988550 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.988494 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:56.988550 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.988511 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:56.989830 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.989809 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-dshm" pod "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" (UID: "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:17:56.989922 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.989902 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-kube-api-access-nx6z8" pod "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" (UID: "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f"). InnerVolumeSpecName "kube-api-access-nx6z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:17:56.989969 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:56.989926 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tls-certs" pod "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" (UID: "1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:17:57.090523 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.090359 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx6z8\" (UniqueName: \"kubernetes.io/projected/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-kube-api-access-nx6z8\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.090523 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.090398 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.090523 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.090417 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:17:57.778154 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.778123 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" event={"ID":"1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f","Type":"ContainerDied","Data":"a54de191549392d12e6b4f1ce16a6ce9b314aac79c15e915d7f2406db3e2b6d0"} Nov 03 04:17:57.778656 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.778157 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq" Nov 03 04:17:57.778656 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.778171 5117 scope.go:122] "RemoveContainer" containerID="d7e2c6c780a7f4251a421fbac5eba26aa70bbd12068e031ae0fb81bbd519b948" Nov 03 04:17:57.946759 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.946723 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq"] Nov 03 04:17:57.950729 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:57.950702 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-w3-68b670-kserve-54f58dc987-zvfmq"] Nov 03 04:17:59.090481 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:17:59.090426 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" path="/var/lib/kubelet/pods/1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f/volumes" Nov 03 04:18:00.010907 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:18:00.010865 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:18:30.011400 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:18:30.011359 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:19:00.011350 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:19:00.011311 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:19:13.860546 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:13.860507 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk"] Nov 03 04:19:13.861009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:13.860706 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" podUID="392157ef-9f8e-48b6-bdcd-da14bce0f10d" containerName="maas-api" containerID="cri-o://c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8" gracePeriod=30 Nov 03 04:19:14.168818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.168793 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 04:19:14.287405 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.287372 5117 generic.go:356] "Generic (PLEG): container finished" podID="392157ef-9f8e-48b6-bdcd-da14bce0f10d" containerID="c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8" exitCode=0 Nov 03 04:19:14.287631 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.287423 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" Nov 03 04:19:14.287631 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.287491 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" event={"ID":"392157ef-9f8e-48b6-bdcd-da14bce0f10d","Type":"ContainerDied","Data":"c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8"} Nov 03 04:19:14.287739 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.287642 5117 scope.go:122] "RemoveContainer" containerID="c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8" Nov 03 04:19:14.287797 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.287571 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk" event={"ID":"392157ef-9f8e-48b6-bdcd-da14bce0f10d","Type":"ContainerDied","Data":"38ce44be34e678e6288d687ffb2cc23b3b534d096b75000e43b16fec9c88a440"} Nov 03 04:19:14.300172 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.300140 5117 scope.go:122] "RemoveContainer" containerID="c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8" Nov 03 04:19:14.300713 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:19:14.300682 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8\": container with ID starting with c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8 not found: ID does not exist" containerID="c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8" Nov 03 04:19:14.300842 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.300726 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8"} err="failed to get container status \"c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8\": rpc error: code = NotFound desc = could not find container \"c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8\": container with ID starting with c50247349b617d6f2e0e11031e850c34df06f42865c13c8f168f395c66cd82a8 not found: ID does not exist" Nov 03 04:19:14.305227 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.305201 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-metrics-certs\") pod \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " Nov 03 04:19:14.305375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.305363 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392157ef-9f8e-48b6-bdcd-da14bce0f10d-kube-api-access-x7mz8\" (UniqueName: \"kubernetes.io/projected/392157ef-9f8e-48b6-bdcd-da14bce0f10d-kube-api-access-x7mz8\") pod \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " Nov 03 04:19:14.305460 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.305393 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392157ef-9f8e-48b6-bdcd-da14bce0f10d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/392157ef-9f8e-48b6-bdcd-da14bce0f10d-openshift-service-ca\") pod \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " Nov 03 04:19:14.305533 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.305508 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls\") pod \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\" (UID: \"392157ef-9f8e-48b6-bdcd-da14bce0f10d\") " Nov 03 04:19:14.306106 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.306074 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392157ef-9f8e-48b6-bdcd-da14bce0f10d-openshift-service-ca" pod "392157ef-9f8e-48b6-bdcd-da14bce0f10d" (UID: "392157ef-9f8e-48b6-bdcd-da14bce0f10d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:14.307711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.307682 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-metrics-certs" pod "392157ef-9f8e-48b6-bdcd-da14bce0f10d" (UID: "392157ef-9f8e-48b6-bdcd-da14bce0f10d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:19:14.308052 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.308033 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls" pod "392157ef-9f8e-48b6-bdcd-da14bce0f10d" (UID: "392157ef-9f8e-48b6-bdcd-da14bce0f10d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:19:14.308163 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.308147 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392157ef-9f8e-48b6-bdcd-da14bce0f10d-kube-api-access-x7mz8" pod "392157ef-9f8e-48b6-bdcd-da14bce0f10d" (UID: "392157ef-9f8e-48b6-bdcd-da14bce0f10d"). InnerVolumeSpecName "kube-api-access-x7mz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:14.407608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.407524 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:14.407608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.407563 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7mz8\" (UniqueName: \"kubernetes.io/projected/392157ef-9f8e-48b6-bdcd-da14bce0f10d-kube-api-access-x7mz8\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:14.407608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.407578 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/392157ef-9f8e-48b6-bdcd-da14bce0f10d-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:14.407608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.407592 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/392157ef-9f8e-48b6-bdcd-da14bce0f10d-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:14.710945 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.710858 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk"] Nov 03 04:19:14.716688 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:14.716659 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-68b670-6497b7455b-xndjk"] Nov 03 04:19:15.090924 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:15.090889 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392157ef-9f8e-48b6-bdcd-da14bce0f10d" path="/var/lib/kubelet/pods/392157ef-9f8e-48b6-bdcd-da14bce0f10d/volumes" Nov 03 04:19:20.103813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:20.103775 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb"] Nov 03 04:19:20.104323 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:20.104030 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" containerID="cri-o://6b27161bfbd0a3be1051379ff91b16c0d2a96a3cb901bb07bd25c29d9bca4109" gracePeriod=30 Nov 03 04:19:25.358289 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.358259 5117 generic.go:356] "Generic (PLEG): container finished" podID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerID="6b27161bfbd0a3be1051379ff91b16c0d2a96a3cb901bb07bd25c29d9bca4109" exitCode=0 Nov 03 04:19:25.358669 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.358329 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" event={"ID":"b2949fa5-406f-4816-af0a-c93db25f9db5","Type":"ContainerDied","Data":"6b27161bfbd0a3be1051379ff91b16c0d2a96a3cb901bb07bd25c29d9bca4109"} Nov 03 04:19:25.379432 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.379371 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 04:19:25.441337 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441303 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-data\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441583 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441350 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-socket\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441583 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441374 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-envoy\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441583 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441396 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-certs\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441583 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441542 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-podinfo\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441676 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-token\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-token\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441707 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2949fa5-406f-4816-af0a-c93db25f9db5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2949fa5-406f-4816-af0a-c93db25f9db5-istiod-ca-cert\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441719 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-socket" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.441818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441742 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-kube-api-access-sjcjc\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-kube-api-access-sjcjc\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.441818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441746 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-data" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.441818 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441755 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-certs" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.442139 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.441858 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-credential-socket\") pod \"b2949fa5-406f-4816-af0a-c93db25f9db5\" (UID: \"b2949fa5-406f-4816-af0a-c93db25f9db5\") " Nov 03 04:19:25.442139 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.442103 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2949fa5-406f-4816-af0a-c93db25f9db5-istiod-ca-cert" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:25.442346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.442310 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-credential-socket" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.442716 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.442696 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2949fa5-406f-4816-af0a-c93db25f9db5-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.442813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.442724 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.442813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.442739 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.442813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.442755 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.442813 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.442770 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.444159 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.444135 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-podinfo" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:19:25.444292 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.444183 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-kube-api-access-sjcjc" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "kube-api-access-sjcjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:25.444551 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.444534 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-envoy" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:19:25.444639 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.444557 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-token" pod "b2949fa5-406f-4816-af0a-c93db25f9db5" (UID: "b2949fa5-406f-4816-af0a-c93db25f9db5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:25.544350 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.544309 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.544350 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.544347 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.544601 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.544364 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:25.544601 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:25.544379 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjcjc\" (UniqueName: \"kubernetes.io/projected/b2949fa5-406f-4816-af0a-c93db25f9db5-kube-api-access-sjcjc\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:26.365757 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:26.365720 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" event={"ID":"b2949fa5-406f-4816-af0a-c93db25f9db5","Type":"ContainerDied","Data":"cdb15beae434a3f7358329445e4f49c8b14a0daa259819d836a8ca2f0c92ac89"} Nov 03 04:19:26.365757 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:26.365740 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb" Nov 03 04:19:26.366332 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:26.365780 5117 scope.go:122] "RemoveContainer" containerID="6b27161bfbd0a3be1051379ff91b16c0d2a96a3cb901bb07bd25c29d9bca4109" Nov 03 04:19:26.586177 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:26.586134 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb"] Nov 03 04:19:26.636494 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:26.636420 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6fprb"] Nov 03 04:19:27.090072 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:27.090039 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" path="/var/lib/kubelet/pods/b2949fa5-406f-4816-af0a-c93db25f9db5/volumes" Nov 03 04:19:30.012008 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:19:30.011967 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:19:50.089756 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.089677 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2"] Nov 03 04:19:50.090305 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.089868 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" podUID="c8070701-d6c6-4399-88d0-298b5fca3efb" containerName="maas-api" containerID="cri-o://d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6" gracePeriod=30 Nov 03 04:19:50.388782 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.388755 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:19:50.413797 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.413769 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8070701-d6c6-4399-88d0-298b5fca3efb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8070701-d6c6-4399-88d0-298b5fca3efb-openshift-service-ca\") pod \"c8070701-d6c6-4399-88d0-298b5fca3efb\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " Nov 03 04:19:50.413935 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.413813 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8070701-d6c6-4399-88d0-298b5fca3efb-kube-api-access-q8ngf\" (UniqueName: \"kubernetes.io/projected/c8070701-d6c6-4399-88d0-298b5fca3efb-kube-api-access-q8ngf\") pod \"c8070701-d6c6-4399-88d0-298b5fca3efb\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " Nov 03 04:19:50.413935 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.413880 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls\") pod \"c8070701-d6c6-4399-88d0-298b5fca3efb\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " Nov 03 04:19:50.414078 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.413994 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-metrics-certs\") pod \"c8070701-d6c6-4399-88d0-298b5fca3efb\" (UID: \"c8070701-d6c6-4399-88d0-298b5fca3efb\") " Nov 03 04:19:50.414187 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.414163 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8070701-d6c6-4399-88d0-298b5fca3efb-openshift-service-ca" pod "c8070701-d6c6-4399-88d0-298b5fca3efb" (UID: "c8070701-d6c6-4399-88d0-298b5fca3efb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:19:50.414521 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.414498 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8070701-d6c6-4399-88d0-298b5fca3efb-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:50.416221 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.416193 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls" pod "c8070701-d6c6-4399-88d0-298b5fca3efb" (UID: "c8070701-d6c6-4399-88d0-298b5fca3efb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:19:50.416340 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.416319 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8070701-d6c6-4399-88d0-298b5fca3efb-kube-api-access-q8ngf" pod "c8070701-d6c6-4399-88d0-298b5fca3efb" (UID: "c8070701-d6c6-4399-88d0-298b5fca3efb"). InnerVolumeSpecName "kube-api-access-q8ngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:19:50.416636 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.416614 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-metrics-certs" pod "c8070701-d6c6-4399-88d0-298b5fca3efb" (UID: "c8070701-d6c6-4399-88d0-298b5fca3efb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:19:50.515686 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.515651 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:50.515686 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.515684 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ngf\" (UniqueName: \"kubernetes.io/projected/c8070701-d6c6-4399-88d0-298b5fca3efb-kube-api-access-q8ngf\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:50.515945 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.515701 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8070701-d6c6-4399-88d0-298b5fca3efb-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:19:50.516777 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.516745 5117 generic.go:356] "Generic (PLEG): container finished" podID="c8070701-d6c6-4399-88d0-298b5fca3efb" containerID="d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6" exitCode=0 Nov 03 04:19:50.516925 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.516791 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" Nov 03 04:19:50.516925 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.516821 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" event={"ID":"c8070701-d6c6-4399-88d0-298b5fca3efb","Type":"ContainerDied","Data":"d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6"} Nov 03 04:19:50.516925 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.516902 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2" event={"ID":"c8070701-d6c6-4399-88d0-298b5fca3efb","Type":"ContainerDied","Data":"314d517672791b9ba4ba96d6dc8b4623d696cce8973a411c0b02f1b3b2548510"} Nov 03 04:19:50.516925 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.516924 5117 scope.go:122] "RemoveContainer" containerID="d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6" Nov 03 04:19:50.528052 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.528007 5117 scope.go:122] "RemoveContainer" containerID="d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6" Nov 03 04:19:50.528402 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:19:50.528378 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6\": container with ID starting with d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6 not found: ID does not exist" containerID="d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6" Nov 03 04:19:50.528529 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.528416 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6"} err="failed to get container status \"d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6\": rpc error: code = NotFound desc = could not find container \"d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6\": container with ID starting with d4ead4267e14e050ecf0ef1677942e87143a96995249eb9c00ec00ffe6e1d0e6 not found: ID does not exist" Nov 03 04:19:50.684806 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.684700 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2"] Nov 03 04:19:50.688650 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:50.688623 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-255c49-67cdd5ff8f-w8xw2"] Nov 03 04:19:51.091112 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:19:51.091075 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8070701-d6c6-4399-88d0-298b5fca3efb" path="/var/lib/kubelet/pods/c8070701-d6c6-4399-88d0-298b5fca3efb/volumes" Nov 03 04:20:00.011313 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:20:00.011272 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:20:21.723825 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:21.723793 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk"] Nov 03 04:20:21.724288 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:21.723997 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" podUID="d4d334d3-ed81-4de0-9422-b9f8e1d123ad" containerName="maas-api" containerID="cri-o://c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4" gracePeriod=30 Nov 03 04:20:22.026574 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.026539 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:20:22.075322 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.075283 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-openshift-service-ca\") pod \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " Nov 03 04:20:22.075532 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.075337 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-kube-api-access-mlz8q\" (UniqueName: \"kubernetes.io/projected/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-kube-api-access-mlz8q\") pod \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " Nov 03 04:20:22.075532 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.075435 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls\") pod \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " Nov 03 04:20:22.075673 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.075579 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-metrics-certs\") pod \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\" (UID: \"d4d334d3-ed81-4de0-9422-b9f8e1d123ad\") " Nov 03 04:20:22.075784 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.075760 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-openshift-service-ca" pod "d4d334d3-ed81-4de0-9422-b9f8e1d123ad" (UID: "d4d334d3-ed81-4de0-9422-b9f8e1d123ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:20:22.076037 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.076016 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:22.077957 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.077929 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls" pod "d4d334d3-ed81-4de0-9422-b9f8e1d123ad" (UID: "d4d334d3-ed81-4de0-9422-b9f8e1d123ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:20:22.077957 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.077946 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-kube-api-access-mlz8q" pod "d4d334d3-ed81-4de0-9422-b9f8e1d123ad" (UID: "d4d334d3-ed81-4de0-9422-b9f8e1d123ad"). InnerVolumeSpecName "kube-api-access-mlz8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:20:22.078195 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.078171 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-metrics-certs" pod "d4d334d3-ed81-4de0-9422-b9f8e1d123ad" (UID: "d4d334d3-ed81-4de0-9422-b9f8e1d123ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:20:22.177190 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.177157 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:22.177190 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.177185 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:22.177190 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.177195 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlz8q\" (UniqueName: \"kubernetes.io/projected/d4d334d3-ed81-4de0-9422-b9f8e1d123ad-kube-api-access-mlz8q\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:20:22.746663 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.746616 5117 generic.go:356] "Generic (PLEG): container finished" podID="d4d334d3-ed81-4de0-9422-b9f8e1d123ad" containerID="c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4" exitCode=0 Nov 03 04:20:22.747171 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.746672 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" Nov 03 04:20:22.747171 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.746704 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" event={"ID":"d4d334d3-ed81-4de0-9422-b9f8e1d123ad","Type":"ContainerDied","Data":"c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4"} Nov 03 04:20:22.747171 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.746765 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk" event={"ID":"d4d334d3-ed81-4de0-9422-b9f8e1d123ad","Type":"ContainerDied","Data":"fffb7f3c156254dd3f6a770cef31e6db13b75d6642d2fca33175c3cd5738d9f8"} Nov 03 04:20:22.747171 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.746794 5117 scope.go:122] "RemoveContainer" containerID="c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4" Nov 03 04:20:22.757821 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.757802 5117 scope.go:122] "RemoveContainer" containerID="c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4" Nov 03 04:20:22.758096 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:20:22.758079 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4\": container with ID starting with c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4 not found: ID does not exist" containerID="c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4" Nov 03 04:20:22.758168 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.758104 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4"} err="failed to get container status \"c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4\": rpc error: code = NotFound desc = could not find container \"c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4\": container with ID starting with c1ec3057dbc13d875a128db1ab30004c60002db337b5bf08773d8ece0eef9ba4 not found: ID does not exist" Nov 03 04:20:22.914773 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.914738 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk"] Nov 03 04:20:22.965327 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:22.965293 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e30496c2-5d79448777-p8srk"] Nov 03 04:20:23.093863 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:20:23.093803 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d334d3-ed81-4de0-9422-b9f8e1d123ad" path="/var/lib/kubelet/pods/d4d334d3-ed81-4de0-9422-b9f8e1d123ad/volumes" Nov 03 04:20:30.024523 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:20:30.024479 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:21:00.011069 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:21:00.011030 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:21:07.845182 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:07.845147 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:21:07.850823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:07.850801 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:21:07.860810 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:07.860791 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:21:07.865879 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:07.865857 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:21:26.996666 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:26.996630 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j"] Nov 03 04:21:26.997125 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:26.996828 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" podUID="51f3a7f5-ca55-471e-b18c-1bd44a8c1198" containerName="maas-api" containerID="cri-o://1f55e600edc883dae90c293d4703398030e43733d0e7ed75734cf9500b5fb2db" gracePeriod=30 Nov 03 04:21:27.162575 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.162538 5117 generic.go:356] "Generic (PLEG): container finished" podID="51f3a7f5-ca55-471e-b18c-1bd44a8c1198" containerID="1f55e600edc883dae90c293d4703398030e43733d0e7ed75734cf9500b5fb2db" exitCode=0 Nov 03 04:21:27.162770 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.162586 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" event={"ID":"51f3a7f5-ca55-471e-b18c-1bd44a8c1198","Type":"ContainerDied","Data":"1f55e600edc883dae90c293d4703398030e43733d0e7ed75734cf9500b5fb2db"} Nov 03 04:21:27.289966 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.289931 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 04:21:27.318363 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.318337 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-openshift-service-ca\") pod \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " Nov 03 04:21:27.318527 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.318397 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-metrics-certs\") pod \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " Nov 03 04:21:27.318527 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.318456 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls\") pod \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " Nov 03 04:21:27.318527 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.318497 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-kube-api-access-8ts7r\" (UniqueName: \"kubernetes.io/projected/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-kube-api-access-8ts7r\") pod \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\" (UID: \"51f3a7f5-ca55-471e-b18c-1bd44a8c1198\") " Nov 03 04:21:27.318789 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.318762 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-openshift-service-ca" pod "51f3a7f5-ca55-471e-b18c-1bd44a8c1198" (UID: "51f3a7f5-ca55-471e-b18c-1bd44a8c1198"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:21:27.318974 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.318953 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:27.321335 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.320857 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-kube-api-access-8ts7r" pod "51f3a7f5-ca55-471e-b18c-1bd44a8c1198" (UID: "51f3a7f5-ca55-471e-b18c-1bd44a8c1198"). InnerVolumeSpecName "kube-api-access-8ts7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:21:27.321335 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.320998 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls" pod "51f3a7f5-ca55-471e-b18c-1bd44a8c1198" (UID: "51f3a7f5-ca55-471e-b18c-1bd44a8c1198"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:21:27.321335 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.321098 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-metrics-certs" pod "51f3a7f5-ca55-471e-b18c-1bd44a8c1198" (UID: "51f3a7f5-ca55-471e-b18c-1bd44a8c1198"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:21:27.420339 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.420298 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:27.420339 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.420335 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:27.420603 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:27.420351 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ts7r\" (UniqueName: \"kubernetes.io/projected/51f3a7f5-ca55-471e-b18c-1bd44a8c1198-kube-api-access-8ts7r\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:21:28.171682 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:28.171655 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" Nov 03 04:21:28.172154 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:28.171654 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j" event={"ID":"51f3a7f5-ca55-471e-b18c-1bd44a8c1198","Type":"ContainerDied","Data":"6b9cb974cadcf3938bd8586575be92849dfb1c94f52ca3b47888d0450b0171d5"} Nov 03 04:21:28.172154 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:28.171799 5117 scope.go:122] "RemoveContainer" containerID="1f55e600edc883dae90c293d4703398030e43733d0e7ed75734cf9500b5fb2db" Nov 03 04:21:28.339191 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:28.339152 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j"] Nov 03 04:21:28.389752 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:28.389715 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-10120d-59988c86cb-5qv2j"] Nov 03 04:21:29.089467 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:21:29.089424 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f3a7f5-ca55-471e-b18c-1bd44a8c1198" path="/var/lib/kubelet/pods/51f3a7f5-ca55-471e-b18c-1bd44a8c1198/volumes" Nov 03 04:21:30.010822 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:21:30.010786 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:22:00.011002 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:22:00.010958 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:22:18.482676 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.482639 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf"] Nov 03 04:22:18.483175 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.482876 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" podUID="b4e5fc07-68de-42ef-8737-354f5fee1710" containerName="maas-api" containerID="cri-o://314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab" gracePeriod=30 Nov 03 04:22:18.787825 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.787797 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 04:22:18.861928 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.861893 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls\") pod \"b4e5fc07-68de-42ef-8737-354f5fee1710\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " Nov 03 04:22:18.862133 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.861961 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e5fc07-68de-42ef-8737-354f5fee1710-kube-api-access-2h482\" (UniqueName: \"kubernetes.io/projected/b4e5fc07-68de-42ef-8737-354f5fee1710-kube-api-access-2h482\") pod \"b4e5fc07-68de-42ef-8737-354f5fee1710\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " Nov 03 04:22:18.862133 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.862033 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-metrics-certs\") pod \"b4e5fc07-68de-42ef-8737-354f5fee1710\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " Nov 03 04:22:18.862133 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.862080 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e5fc07-68de-42ef-8737-354f5fee1710-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e5fc07-68de-42ef-8737-354f5fee1710-openshift-service-ca\") pod \"b4e5fc07-68de-42ef-8737-354f5fee1710\" (UID: \"b4e5fc07-68de-42ef-8737-354f5fee1710\") " Nov 03 04:22:18.863075 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.863017 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e5fc07-68de-42ef-8737-354f5fee1710-openshift-service-ca" pod "b4e5fc07-68de-42ef-8737-354f5fee1710" (UID: "b4e5fc07-68de-42ef-8737-354f5fee1710"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:18.863979 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.863950 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e5fc07-68de-42ef-8737-354f5fee1710-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:18.864824 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.864788 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-metrics-certs" pod "b4e5fc07-68de-42ef-8737-354f5fee1710" (UID: "b4e5fc07-68de-42ef-8737-354f5fee1710"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:18.865048 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.864992 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls" pod "b4e5fc07-68de-42ef-8737-354f5fee1710" (UID: "b4e5fc07-68de-42ef-8737-354f5fee1710"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:18.865048 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.865035 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e5fc07-68de-42ef-8737-354f5fee1710-kube-api-access-2h482" pod "b4e5fc07-68de-42ef-8737-354f5fee1710" (UID: "b4e5fc07-68de-42ef-8737-354f5fee1710"). InnerVolumeSpecName "kube-api-access-2h482". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:18.965339 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.965306 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:18.965339 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.965335 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h482\" (UniqueName: \"kubernetes.io/projected/b4e5fc07-68de-42ef-8737-354f5fee1710-kube-api-access-2h482\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:18.965339 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:18.965345 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4e5fc07-68de-42ef-8737-354f5fee1710-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:19.474414 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.474310 5117 generic.go:356] "Generic (PLEG): container finished" podID="b4e5fc07-68de-42ef-8737-354f5fee1710" containerID="314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab" exitCode=0 Nov 03 04:22:19.474414 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.474360 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" Nov 03 04:22:19.474650 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.474414 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" event={"ID":"b4e5fc07-68de-42ef-8737-354f5fee1710","Type":"ContainerDied","Data":"314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab"} Nov 03 04:22:19.474650 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.474516 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf" event={"ID":"b4e5fc07-68de-42ef-8737-354f5fee1710","Type":"ContainerDied","Data":"dbc3a3d098be26354ca718546d1da016522360c4882b379a20d48e71d294fbc1"} Nov 03 04:22:19.474650 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.474546 5117 scope.go:122] "RemoveContainer" containerID="314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab" Nov 03 04:22:19.485222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.484981 5117 scope.go:122] "RemoveContainer" containerID="314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab" Nov 03 04:22:19.485526 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:22:19.485280 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab\": container with ID starting with 314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab not found: ID does not exist" containerID="314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab" Nov 03 04:22:19.485526 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.485319 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab"} err="failed to get container status \"314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab\": rpc error: code = NotFound desc = could not find container \"314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab\": container with ID starting with 314357e7ccdfefc529405fa1decd8a7772ad2a8d57e4162d58edfb845fa684ab not found: ID does not exist" Nov 03 04:22:19.642880 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.642850 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf"] Nov 03 04:22:19.645225 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.645198 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7ed2d6-7cd9ccdd8b-p6trf"] Nov 03 04:22:19.694047 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.694013 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct"] Nov 03 04:22:19.694267 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.694243 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" podUID="47833055-fc67-4320-b300-05193b9dd528" containerName="maas-api" containerID="cri-o://37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4" gracePeriod=30 Nov 03 04:22:19.989793 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:19.989762 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 04:22:20.078418 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.078335 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls\") pod \"47833055-fc67-4320-b300-05193b9dd528\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " Nov 03 04:22:20.078625 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.078438 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47833055-fc67-4320-b300-05193b9dd528-kube-api-access-m4czg\" (UniqueName: \"kubernetes.io/projected/47833055-fc67-4320-b300-05193b9dd528-kube-api-access-m4czg\") pod \"47833055-fc67-4320-b300-05193b9dd528\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " Nov 03 04:22:20.078625 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.078522 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-metrics-certs\") pod \"47833055-fc67-4320-b300-05193b9dd528\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " Nov 03 04:22:20.078625 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.078555 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47833055-fc67-4320-b300-05193b9dd528-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47833055-fc67-4320-b300-05193b9dd528-openshift-service-ca\") pod \"47833055-fc67-4320-b300-05193b9dd528\" (UID: \"47833055-fc67-4320-b300-05193b9dd528\") " Nov 03 04:22:20.078995 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.078965 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47833055-fc67-4320-b300-05193b9dd528-openshift-service-ca" pod "47833055-fc67-4320-b300-05193b9dd528" (UID: "47833055-fc67-4320-b300-05193b9dd528"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:22:20.080887 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.080855 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47833055-fc67-4320-b300-05193b9dd528-kube-api-access-m4czg" pod "47833055-fc67-4320-b300-05193b9dd528" (UID: "47833055-fc67-4320-b300-05193b9dd528"). InnerVolumeSpecName "kube-api-access-m4czg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:20.081068 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.081051 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls" pod "47833055-fc67-4320-b300-05193b9dd528" (UID: "47833055-fc67-4320-b300-05193b9dd528"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:20.081180 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.081159 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-metrics-certs" pod "47833055-fc67-4320-b300-05193b9dd528" (UID: "47833055-fc67-4320-b300-05193b9dd528"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:20.180514 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.180475 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:20.180514 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.180514 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4czg\" (UniqueName: \"kubernetes.io/projected/47833055-fc67-4320-b300-05193b9dd528-kube-api-access-m4czg\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:20.180716 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.180529 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47833055-fc67-4320-b300-05193b9dd528-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:20.180716 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.180543 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47833055-fc67-4320-b300-05193b9dd528-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:20.482114 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.482029 5117 generic.go:356] "Generic (PLEG): container finished" podID="47833055-fc67-4320-b300-05193b9dd528" containerID="37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4" exitCode=0 Nov 03 04:22:20.482114 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.482079 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" Nov 03 04:22:20.482325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.482117 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" event={"ID":"47833055-fc67-4320-b300-05193b9dd528","Type":"ContainerDied","Data":"37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4"} Nov 03 04:22:20.482325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.482171 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct" event={"ID":"47833055-fc67-4320-b300-05193b9dd528","Type":"ContainerDied","Data":"594e4532ed9132418984ed5402cdc2b19fbe4797ae333261b5177e865dc50edc"} Nov 03 04:22:20.482325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.482197 5117 scope.go:122] "RemoveContainer" containerID="37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4" Nov 03 04:22:20.497651 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.497402 5117 scope.go:122] "RemoveContainer" containerID="37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4" Nov 03 04:22:20.497949 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:22:20.497727 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4\": container with ID starting with 37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4 not found: ID does not exist" containerID="37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4" Nov 03 04:22:20.497949 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.497764 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4"} err="failed to get container status \"37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4\": rpc error: code = NotFound desc = could not find container \"37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4\": container with ID starting with 37b84b7bd054e02f28f58ada25a4232512bbc0dfe4d3e9a7450aa5a5057da2d4 not found: ID does not exist" Nov 03 04:22:20.650233 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.650198 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct"] Nov 03 04:22:20.654594 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:20.654567 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-9daccc-9dcd45b4d-wvnct"] Nov 03 04:22:21.091060 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:21.091029 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47833055-fc67-4320-b300-05193b9dd528" path="/var/lib/kubelet/pods/47833055-fc67-4320-b300-05193b9dd528/volumes" Nov 03 04:22:21.091415 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:21.091402 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e5fc07-68de-42ef-8737-354f5fee1710" path="/var/lib/kubelet/pods/b4e5fc07-68de-42ef-8737-354f5fee1710/volumes" Nov 03 04:22:30.012240 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:22:30.012193 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:22:33.547117 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:33.547083 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs"] Nov 03 04:22:48.572332 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.572210 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" containerID="cri-o://a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa" gracePeriod=45 Nov 03 04:22:48.830507 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.830432 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 04:22:48.862492 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862438 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-home\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-home\") pod \"8333631b-54aa-4041-a8c6-a725077a336f\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " Nov 03 04:22:48.862653 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862542 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8333631b-54aa-4041-a8c6-a725077a336f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8333631b-54aa-4041-a8c6-a725077a336f-tls-certs\") pod \"8333631b-54aa-4041-a8c6-a725077a336f\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " Nov 03 04:22:48.862653 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862579 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-tmp-dir\") pod \"8333631b-54aa-4041-a8c6-a725077a336f\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " Nov 03 04:22:48.862653 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862627 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-model-cache\") pod \"8333631b-54aa-4041-a8c6-a725077a336f\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " Nov 03 04:22:48.862817 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862701 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-home" pod "8333631b-54aa-4041-a8c6-a725077a336f" (UID: "8333631b-54aa-4041-a8c6-a725077a336f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:48.862817 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862724 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-dshm\") pod \"8333631b-54aa-4041-a8c6-a725077a336f\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " Nov 03 04:22:48.862817 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862768 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8333631b-54aa-4041-a8c6-a725077a336f-kube-api-access-cq5jg\" (UniqueName: \"kubernetes.io/projected/8333631b-54aa-4041-a8c6-a725077a336f-kube-api-access-cq5jg\") pod \"8333631b-54aa-4041-a8c6-a725077a336f\" (UID: \"8333631b-54aa-4041-a8c6-a725077a336f\") " Nov 03 04:22:48.862971 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862876 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-tmp-dir" pod "8333631b-54aa-4041-a8c6-a725077a336f" (UID: "8333631b-54aa-4041-a8c6-a725077a336f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:48.862971 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.862888 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-model-cache" pod "8333631b-54aa-4041-a8c6-a725077a336f" (UID: "8333631b-54aa-4041-a8c6-a725077a336f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:48.863181 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.863157 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:48.863181 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.863183 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:48.863354 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.863198 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:48.864991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.864969 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-dshm" pod "8333631b-54aa-4041-a8c6-a725077a336f" (UID: "8333631b-54aa-4041-a8c6-a725077a336f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:22:48.865105 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.865014 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8333631b-54aa-4041-a8c6-a725077a336f-kube-api-access-cq5jg" pod "8333631b-54aa-4041-a8c6-a725077a336f" (UID: "8333631b-54aa-4041-a8c6-a725077a336f"). InnerVolumeSpecName "kube-api-access-cq5jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:22:48.865105 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.865049 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8333631b-54aa-4041-a8c6-a725077a336f-tls-certs" pod "8333631b-54aa-4041-a8c6-a725077a336f" (UID: "8333631b-54aa-4041-a8c6-a725077a336f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:22:48.964073 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.964041 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8333631b-54aa-4041-a8c6-a725077a336f-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:48.964073 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.964067 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8333631b-54aa-4041-a8c6-a725077a336f-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:48.964073 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:48.964077 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5jg\" (UniqueName: \"kubernetes.io/projected/8333631b-54aa-4041-a8c6-a725077a336f-kube-api-access-cq5jg\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:22:49.667281 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.667241 5117 generic.go:356] "Generic (PLEG): container finished" podID="8333631b-54aa-4041-a8c6-a725077a336f" containerID="a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa" exitCode=0 Nov 03 04:22:49.667760 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.667292 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" Nov 03 04:22:49.667760 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.667319 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" event={"ID":"8333631b-54aa-4041-a8c6-a725077a336f","Type":"ContainerDied","Data":"a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa"} Nov 03 04:22:49.667760 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.667370 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" event={"ID":"8333631b-54aa-4041-a8c6-a725077a336f","Type":"ContainerDied","Data":"a2a11e6cc64af4038a73ceb8ab8bccd5688fa6699b2ba8eedcfd67522ff7db94"} Nov 03 04:22:49.667760 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.667391 5117 scope.go:122] "RemoveContainer" containerID="a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa" Nov 03 04:22:49.676748 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.676731 5117 scope.go:122] "RemoveContainer" containerID="a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa" Nov 03 04:22:49.677022 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:22:49.677003 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa\": container with ID starting with a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa not found: ID does not exist" containerID="a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa" Nov 03 04:22:49.677068 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.677031 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa"} err="failed to get container status \"a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa\": rpc error: code = NotFound desc = could not find container \"a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa\": container with ID starting with a8e521eb40a0b94bdaeb7d3e363e6fec61dc4c87e0e01c07252c7ed2ae578efa not found: ID does not exist" Nov 03 04:22:49.816721 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.816676 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 04:22:49.836020 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.835988 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs"] Nov 03 04:22:49.840405 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:49.840382 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/premium-w1-5a6246-kserve-d4564db9-9nffs"] Nov 03 04:22:51.089290 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:22:51.089256 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8333631b-54aa-4041-a8c6-a725077a336f" path="/var/lib/kubelet/pods/8333631b-54aa-4041-a8c6-a725077a336f/volumes" Nov 03 04:23:00.011131 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:23:00.011092 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:23:25.900754 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:25.900715 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g"] Nov 03 04:23:25.901213 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:25.900931 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" podUID="73a3d918-5836-4a33-b411-a05af4647b7b" containerName="maas-api" containerID="cri-o://cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee" gracePeriod=30 Nov 03 04:23:26.032845 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.032802 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" podUID="73a3d918-5836-4a33-b411-a05af4647b7b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": dial tcp 10.132.0.58:8443: connect: connection refused" Nov 03 04:23:26.190758 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.190730 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 04:23:26.251581 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.251554 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls\") pod \"73a3d918-5836-4a33-b411-a05af4647b7b\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " Nov 03 04:23:26.251782 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.251606 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a3d918-5836-4a33-b411-a05af4647b7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a3d918-5836-4a33-b411-a05af4647b7b-openshift-service-ca\") pod \"73a3d918-5836-4a33-b411-a05af4647b7b\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " Nov 03 04:23:26.251782 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.251763 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-metrics-certs\") pod \"73a3d918-5836-4a33-b411-a05af4647b7b\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " Nov 03 04:23:26.251904 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.251798 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a3d918-5836-4a33-b411-a05af4647b7b-kube-api-access-9kr8q\" (UniqueName: \"kubernetes.io/projected/73a3d918-5836-4a33-b411-a05af4647b7b-kube-api-access-9kr8q\") pod \"73a3d918-5836-4a33-b411-a05af4647b7b\" (UID: \"73a3d918-5836-4a33-b411-a05af4647b7b\") " Nov 03 04:23:26.252074 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.252036 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a3d918-5836-4a33-b411-a05af4647b7b-openshift-service-ca" pod "73a3d918-5836-4a33-b411-a05af4647b7b" (UID: "73a3d918-5836-4a33-b411-a05af4647b7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:26.252213 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.252194 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a3d918-5836-4a33-b411-a05af4647b7b-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:26.253991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.253959 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-metrics-certs" pod "73a3d918-5836-4a33-b411-a05af4647b7b" (UID: "73a3d918-5836-4a33-b411-a05af4647b7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:23:26.254107 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.254042 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls" pod "73a3d918-5836-4a33-b411-a05af4647b7b" (UID: "73a3d918-5836-4a33-b411-a05af4647b7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:23:26.254177 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.254114 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a3d918-5836-4a33-b411-a05af4647b7b-kube-api-access-9kr8q" pod "73a3d918-5836-4a33-b411-a05af4647b7b" (UID: "73a3d918-5836-4a33-b411-a05af4647b7b"). InnerVolumeSpecName "kube-api-access-9kr8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:26.353814 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.353770 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:26.353814 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.353801 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kr8q\" (UniqueName: \"kubernetes.io/projected/73a3d918-5836-4a33-b411-a05af4647b7b-kube-api-access-9kr8q\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:26.353814 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.353811 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a3d918-5836-4a33-b411-a05af4647b7b-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:26.882739 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.882699 5117 generic.go:356] "Generic (PLEG): container finished" podID="73a3d918-5836-4a33-b411-a05af4647b7b" containerID="cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee" exitCode=0 Nov 03 04:23:26.882930 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.882753 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" Nov 03 04:23:26.882930 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.882784 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" event={"ID":"73a3d918-5836-4a33-b411-a05af4647b7b","Type":"ContainerDied","Data":"cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee"} Nov 03 04:23:26.882930 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.882843 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g" event={"ID":"73a3d918-5836-4a33-b411-a05af4647b7b","Type":"ContainerDied","Data":"3f3ace1939f7653b3cb20413ac78efb742e508eb6fbb5146511f1142ebc64b76"} Nov 03 04:23:26.882930 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.882866 5117 scope.go:122] "RemoveContainer" containerID="cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee" Nov 03 04:23:26.893168 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.893143 5117 scope.go:122] "RemoveContainer" containerID="cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee" Nov 03 04:23:26.893463 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:23:26.893422 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee\": container with ID starting with cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee not found: ID does not exist" containerID="cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee" Nov 03 04:23:26.893524 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:26.893473 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee"} err="failed to get container status \"cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee\": rpc error: code = NotFound desc = could not find container \"cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee\": container with ID starting with cf852e7015e9586ef8f93823b3c270b201ef06673ced9db165f634f98dcf80ee not found: ID does not exist" Nov 03 04:23:27.051255 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:27.051211 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g"] Nov 03 04:23:27.102065 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:27.102028 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5a6246-88d45bd9f-5wv5g"] Nov 03 04:23:29.098676 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:29.098638 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a3d918-5836-4a33-b411-a05af4647b7b" path="/var/lib/kubelet/pods/73a3d918-5836-4a33-b411-a05af4647b7b/volumes" Nov 03 04:23:30.013721 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:23:30.013640 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:23:32.177206 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.177172 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc"] Nov 03 04:23:32.177639 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.177421 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" podUID="6b43ac6c-1194-4622-b271-a669efeb6588" containerName="maas-api" containerID="cri-o://7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e" gracePeriod=30 Nov 03 04:23:32.455689 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.455652 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 04:23:32.518581 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.518542 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b43ac6c-1194-4622-b271-a669efeb6588-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43ac6c-1194-4622-b271-a669efeb6588-openshift-service-ca\") pod \"6b43ac6c-1194-4622-b271-a669efeb6588\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " Nov 03 04:23:32.518779 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.518633 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-metrics-certs\") pod \"6b43ac6c-1194-4622-b271-a669efeb6588\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " Nov 03 04:23:32.518779 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.518715 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls\") pod \"6b43ac6c-1194-4622-b271-a669efeb6588\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " Nov 03 04:23:32.518900 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.518798 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b43ac6c-1194-4622-b271-a669efeb6588-kube-api-access-dvz7x\" (UniqueName: \"kubernetes.io/projected/6b43ac6c-1194-4622-b271-a669efeb6588-kube-api-access-dvz7x\") pod \"6b43ac6c-1194-4622-b271-a669efeb6588\" (UID: \"6b43ac6c-1194-4622-b271-a669efeb6588\") " Nov 03 04:23:32.518953 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.518915 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b43ac6c-1194-4622-b271-a669efeb6588-openshift-service-ca" pod "6b43ac6c-1194-4622-b271-a669efeb6588" (UID: "6b43ac6c-1194-4622-b271-a669efeb6588"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:32.519269 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.519230 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b43ac6c-1194-4622-b271-a669efeb6588-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:32.521124 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.521092 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls" pod "6b43ac6c-1194-4622-b271-a669efeb6588" (UID: "6b43ac6c-1194-4622-b271-a669efeb6588"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:23:32.521396 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.521368 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-metrics-certs" pod "6b43ac6c-1194-4622-b271-a669efeb6588" (UID: "6b43ac6c-1194-4622-b271-a669efeb6588"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:23:32.521522 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.521492 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b43ac6c-1194-4622-b271-a669efeb6588-kube-api-access-dvz7x" pod "6b43ac6c-1194-4622-b271-a669efeb6588" (UID: "6b43ac6c-1194-4622-b271-a669efeb6588"). InnerVolumeSpecName "kube-api-access-dvz7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:32.620852 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.620820 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:32.620852 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.620851 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b43ac6c-1194-4622-b271-a669efeb6588-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:32.620852 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.620861 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvz7x\" (UniqueName: \"kubernetes.io/projected/6b43ac6c-1194-4622-b271-a669efeb6588-kube-api-access-dvz7x\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:32.919713 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.919681 5117 generic.go:356] "Generic (PLEG): container finished" podID="6b43ac6c-1194-4622-b271-a669efeb6588" containerID="7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e" exitCode=0 Nov 03 04:23:32.919894 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.919727 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" Nov 03 04:23:32.919894 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.919728 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" event={"ID":"6b43ac6c-1194-4622-b271-a669efeb6588","Type":"ContainerDied","Data":"7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e"} Nov 03 04:23:32.919894 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.919855 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc" event={"ID":"6b43ac6c-1194-4622-b271-a669efeb6588","Type":"ContainerDied","Data":"599c25b590998535e762c558b8f5612c8eef38fbc07978ac4ff7313fdc611c1f"} Nov 03 04:23:32.919894 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.919891 5117 scope.go:122] "RemoveContainer" containerID="7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e" Nov 03 04:23:32.930761 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.930743 5117 scope.go:122] "RemoveContainer" containerID="7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e" Nov 03 04:23:32.931050 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:23:32.931031 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e\": container with ID starting with 7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e not found: ID does not exist" containerID="7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e" Nov 03 04:23:32.931099 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:32.931060 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e"} err="failed to get container status \"7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e\": rpc error: code = NotFound desc = could not find container \"7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e\": container with ID starting with 7bcf02a07734799733fd34f70eeecf713a85db42dbfefce5b664129fdfce840e not found: ID does not exist" Nov 03 04:23:33.090214 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:33.090185 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc"] Nov 03 04:23:33.094855 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:33.094830 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b1ad4e-659dcf9589-kvlgc"] Nov 03 04:23:35.089521 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:35.089487 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b43ac6c-1194-4622-b271-a669efeb6588" path="/var/lib/kubelet/pods/6b43ac6c-1194-4622-b271-a669efeb6588/volumes" Nov 03 04:23:43.584857 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:43.584815 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g"] Nov 03 04:23:43.585365 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:43.585060 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" containerID="cri-o://3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697" gracePeriod=30 Nov 03 04:23:47.348396 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:47.348361 5117 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:23:47.348822 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:47.348429 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:23:48.844338 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:48.844312 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 04:23:49.010330 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.010238 5117 generic.go:356] "Generic (PLEG): container finished" podID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerID="3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697" exitCode=0 Nov 03 04:23:49.010330 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.010275 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" event={"ID":"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94","Type":"ContainerDied","Data":"3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697"} Nov 03 04:23:49.010561 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.010342 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" event={"ID":"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94","Type":"ContainerDied","Data":"afdc0117a5450c7fce814c6fc1affe3eeb28a9f7dcd0f451c82f41ef08657efd"} Nov 03 04:23:49.010561 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.010290 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g" Nov 03 04:23:49.010561 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.010370 5117 scope.go:122] "RemoveContainer" containerID="3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697" Nov 03 04:23:49.014840 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.014815 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-podinfo\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.014958 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.014883 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-data\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.014958 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.014917 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istiod-ca-cert\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.014958 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.014939 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-envoy\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.015131 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.014970 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-credential-socket\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.015131 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015013 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-token\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-token\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.015131 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015052 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-certs\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.015131 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015094 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-socket\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.015344 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015139 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-kube-api-access-m9fhf\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-kube-api-access-m9fhf\") pod \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\" (UID: \"f8d8f94f-9b7f-450e-8bbb-b42776fd7f94\") " Nov 03 04:23:49.015344 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015216 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-data" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.015344 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015258 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-credential-socket" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.015715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015362 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-certs" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.015715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015511 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-socket" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.015715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015629 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.015715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015650 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.015715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015666 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.015715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015681 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.016052 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.015902 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istiod-ca-cert" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:23:49.017750 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.017724 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-podinfo" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:23:49.017882 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.017765 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-envoy" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:23:49.018566 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.018530 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-kube-api-access-m9fhf" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "kube-api-access-m9fhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:49.019588 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.019069 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-token" pod "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" (UID: "f8d8f94f-9b7f-450e-8bbb-b42776fd7f94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:23:49.024032 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.024014 5117 scope.go:122] "RemoveContainer" containerID="3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697" Nov 03 04:23:49.024373 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:23:49.024356 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697\": container with ID starting with 3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697 not found: ID does not exist" containerID="3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697" Nov 03 04:23:49.024464 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.024382 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697"} err="failed to get container status \"3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697\": rpc error: code = NotFound desc = could not find container \"3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697\": container with ID starting with 3e5f5b758a4c6b27c85847f1228d71710a3dac61c0de94b1ff6b0465498c7697 not found: ID does not exist" Nov 03 04:23:49.117107 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.117068 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.117107 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.117106 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fhf\" (UniqueName: \"kubernetes.io/projected/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-kube-api-access-m9fhf\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.117285 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.117122 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.117285 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.117136 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.117285 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.117150 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:23:49.429328 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.429296 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g"] Nov 03 04:23:49.433591 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:49.433566 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-2fp4g"] Nov 03 04:23:51.090066 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:23:51.090028 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" path="/var/lib/kubelet/pods/f8d8f94f-9b7f-450e-8bbb-b42776fd7f94/volumes" Nov 03 04:24:00.011198 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:24:00.011154 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:24:08.021191 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:08.021103 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw"] Nov 03 04:24:08.021711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:08.021300 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" containerID="cri-o://2f333aa39d8d83e2972d1464a9cd82e82554990d8da684d882a81405a66b91ef" gracePeriod=30 Nov 03 04:24:12.985465 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:12.985410 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lx4vw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 04:24:12.985927 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:12.985501 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 04:24:13.156143 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.156088 5117 generic.go:356] "Generic (PLEG): container finished" podID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerID="2f333aa39d8d83e2972d1464a9cd82e82554990d8da684d882a81405a66b91ef" exitCode=0 Nov 03 04:24:13.156318 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.156164 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" event={"ID":"11e1aa63-6b48-40dd-ad68-ef76b84e6374","Type":"ContainerDied","Data":"2f333aa39d8d83e2972d1464a9cd82e82554990d8da684d882a81405a66b91ef"} Nov 03 04:24:13.282792 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.282755 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:24:13.366337 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366302 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-certs\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366337 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366340 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-credential-socket\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366620 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366366 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-data\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366620 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366479 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-podinfo\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366620 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366540 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-token\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-token\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366620 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366597 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-kube-api-access-h8mdw\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-kube-api-access-h8mdw\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366830 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366650 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-socket\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366830 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366650 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-credential-socket" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.366830 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366685 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istiod-ca-cert\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366830 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366691 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-certs" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.366830 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366717 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-envoy\") pod \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\" (UID: \"11e1aa63-6b48-40dd-ad68-ef76b84e6374\") " Nov 03 04:24:13.366830 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.366786 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-data" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.367126 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.367050 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istiod-ca-cert" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:13.367190 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.367150 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-socket" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.367413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.367217 5117 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.367413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.367237 5117 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istiod-ca-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.367413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.367251 5117 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-workload-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.367413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.367296 5117 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-credential-socket\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.367413 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.367307 5117 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-data\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.369021 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.368997 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-envoy" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:13.369121 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.369028 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-podinfo" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 04:24:13.369271 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.369250 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-token" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:13.369335 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.369283 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-kube-api-access-h8mdw" pod "11e1aa63-6b48-40dd-ad68-ef76b84e6374" (UID: "11e1aa63-6b48-40dd-ad68-ef76b84e6374"). InnerVolumeSpecName "kube-api-access-h8mdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:13.468043 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.468013 5117 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-token\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.468043 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.468040 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8mdw\" (UniqueName: \"kubernetes.io/projected/11e1aa63-6b48-40dd-ad68-ef76b84e6374-kube-api-access-h8mdw\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.468043 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.468050 5117 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-envoy\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:13.468287 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:13.468059 5117 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e1aa63-6b48-40dd-ad68-ef76b84e6374-istio-podinfo\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:14.163074 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:14.163032 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" event={"ID":"11e1aa63-6b48-40dd-ad68-ef76b84e6374","Type":"ContainerDied","Data":"4bd5df4a8a6a6b49d4b0249adda2ac127f393c112c2b5c84df66b00fea65bc6a"} Nov 03 04:24:14.163545 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:14.163100 5117 scope.go:122] "RemoveContainer" containerID="2f333aa39d8d83e2972d1464a9cd82e82554990d8da684d882a81405a66b91ef" Nov 03 04:24:14.163545 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:14.163047 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw" Nov 03 04:24:14.331784 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:14.331751 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw"] Nov 03 04:24:14.383026 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:14.382994 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lx4vw"] Nov 03 04:24:15.090404 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:15.090370 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" path="/var/lib/kubelet/pods/11e1aa63-6b48-40dd-ad68-ef76b84e6374/volumes" Nov 03 04:24:25.154823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.154786 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc"] Nov 03 04:24:25.155256 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.155000 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" podUID="5294bb78-bd86-477b-b45e-d211e6cfd02f" containerName="maas-api" containerID="cri-o://1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e" gracePeriod=30 Nov 03 04:24:25.441793 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.441762 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 04:24:25.492197 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.492164 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5294bb78-bd86-477b-b45e-d211e6cfd02f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294bb78-bd86-477b-b45e-d211e6cfd02f-openshift-service-ca\") pod \"5294bb78-bd86-477b-b45e-d211e6cfd02f\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " Nov 03 04:24:25.492356 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.492275 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-metrics-certs\") pod \"5294bb78-bd86-477b-b45e-d211e6cfd02f\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " Nov 03 04:24:25.492356 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.492305 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-maas-api-tls\") pod \"5294bb78-bd86-477b-b45e-d211e6cfd02f\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " Nov 03 04:24:25.492433 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.492360 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5294bb78-bd86-477b-b45e-d211e6cfd02f-kube-api-access-h982c\" (UniqueName: \"kubernetes.io/projected/5294bb78-bd86-477b-b45e-d211e6cfd02f-kube-api-access-h982c\") pod \"5294bb78-bd86-477b-b45e-d211e6cfd02f\" (UID: \"5294bb78-bd86-477b-b45e-d211e6cfd02f\") " Nov 03 04:24:25.492574 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.492531 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5294bb78-bd86-477b-b45e-d211e6cfd02f-openshift-service-ca" pod "5294bb78-bd86-477b-b45e-d211e6cfd02f" (UID: "5294bb78-bd86-477b-b45e-d211e6cfd02f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:24:25.492783 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.492752 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5294bb78-bd86-477b-b45e-d211e6cfd02f-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:25.494799 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.494773 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-maas-api-tls" pod "5294bb78-bd86-477b-b45e-d211e6cfd02f" (UID: "5294bb78-bd86-477b-b45e-d211e6cfd02f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:25.494903 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.494836 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-metrics-certs" pod "5294bb78-bd86-477b-b45e-d211e6cfd02f" (UID: "5294bb78-bd86-477b-b45e-d211e6cfd02f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:25.495073 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.495056 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5294bb78-bd86-477b-b45e-d211e6cfd02f-kube-api-access-h982c" pod "5294bb78-bd86-477b-b45e-d211e6cfd02f" (UID: "5294bb78-bd86-477b-b45e-d211e6cfd02f"). InnerVolumeSpecName "kube-api-access-h982c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:25.593978 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.593934 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h982c\" (UniqueName: \"kubernetes.io/projected/5294bb78-bd86-477b-b45e-d211e6cfd02f-kube-api-access-h982c\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:25.593978 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.593973 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:25.593978 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:25.593987 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5294bb78-bd86-477b-b45e-d211e6cfd02f-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:26.236486 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.236417 5117 generic.go:356] "Generic (PLEG): container finished" podID="5294bb78-bd86-477b-b45e-d211e6cfd02f" containerID="1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e" exitCode=0 Nov 03 04:24:26.236922 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.236478 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" event={"ID":"5294bb78-bd86-477b-b45e-d211e6cfd02f","Type":"ContainerDied","Data":"1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e"} Nov 03 04:24:26.236922 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.236502 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" Nov 03 04:24:26.236922 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.236542 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc" event={"ID":"5294bb78-bd86-477b-b45e-d211e6cfd02f","Type":"ContainerDied","Data":"25dfab78d4eec992c6df56b1a80302cbb11154700a1813cde5a8a32c1462f9c2"} Nov 03 04:24:26.236922 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.236566 5117 scope.go:122] "RemoveContainer" containerID="1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e" Nov 03 04:24:26.247466 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.247432 5117 scope.go:122] "RemoveContainer" containerID="1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e" Nov 03 04:24:26.247713 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:24:26.247694 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e\": container with ID starting with 1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e not found: ID does not exist" containerID="1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e" Nov 03 04:24:26.247752 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.247723 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e"} err="failed to get container status \"1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e\": rpc error: code = NotFound desc = could not find container \"1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e\": container with ID starting with 1d1ea80302d5e4e8f6600ca0a4816cfc451a44babe9ad8ff5e75d15f720c954e not found: ID does not exist" Nov 03 04:24:26.404757 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.404722 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc"] Nov 03 04:24:26.420331 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:26.420299 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-286a8c-589c9946d4-5qtcc"] Nov 03 04:24:27.091273 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:27.091238 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5294bb78-bd86-477b-b45e-d211e6cfd02f" path="/var/lib/kubelet/pods/5294bb78-bd86-477b-b45e-d211e6cfd02f/volumes" Nov 03 04:24:30.021207 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:24:30.021167 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:24:33.332143 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:33.332095 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf"] Nov 03 04:24:34.861645 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:34.861606 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p"] Nov 03 04:24:34.912229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:34.912194 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh"] Nov 03 04:24:48.356100 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.356041 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" podUID="3f3f800d-e685-42ae-b7a8-df2ede6cb623" containerName="main" containerID="cri-o://316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07" gracePeriod=45 Nov 03 04:24:48.498406 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.498369 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" podUID="3f3f800d-e685-42ae-b7a8-df2ede6cb623" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": dial tcp 10.132.0.143:8000: connect: connection refused" Nov 03 04:24:48.626912 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.626887 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:24:48.737553 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737518 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-dshm\") pod \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " Nov 03 04:24:48.737750 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737581 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3f800d-e685-42ae-b7a8-df2ede6cb623-kube-api-access-7dzqp\" (UniqueName: \"kubernetes.io/projected/3f3f800d-e685-42ae-b7a8-df2ede6cb623-kube-api-access-7dzqp\") pod \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " Nov 03 04:24:48.737750 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737643 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tmp-dir\") pod \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " Nov 03 04:24:48.737750 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737674 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-home\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-home\") pod \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " Nov 03 04:24:48.737750 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737732 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-model-cache\") pod \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " Nov 03 04:24:48.737973 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737813 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tls-certs\") pod \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\" (UID: \"3f3f800d-e685-42ae-b7a8-df2ede6cb623\") " Nov 03 04:24:48.737973 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737955 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-home" pod "3f3f800d-e685-42ae-b7a8-df2ede6cb623" (UID: "3f3f800d-e685-42ae-b7a8-df2ede6cb623"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:48.738077 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737973 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tmp-dir" pod "3f3f800d-e685-42ae-b7a8-df2ede6cb623" (UID: "3f3f800d-e685-42ae-b7a8-df2ede6cb623"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:48.738077 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.737997 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-model-cache" pod "3f3f800d-e685-42ae-b7a8-df2ede6cb623" (UID: "3f3f800d-e685-42ae-b7a8-df2ede6cb623"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:48.738222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.738202 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:48.738259 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.738228 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:48.738259 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.738241 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:48.739939 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.739920 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-dshm" pod "3f3f800d-e685-42ae-b7a8-df2ede6cb623" (UID: "3f3f800d-e685-42ae-b7a8-df2ede6cb623"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:48.740028 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.739985 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3f800d-e685-42ae-b7a8-df2ede6cb623-kube-api-access-7dzqp" pod "3f3f800d-e685-42ae-b7a8-df2ede6cb623" (UID: "3f3f800d-e685-42ae-b7a8-df2ede6cb623"). InnerVolumeSpecName "kube-api-access-7dzqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:48.740073 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.740022 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tls-certs" pod "3f3f800d-e685-42ae-b7a8-df2ede6cb623" (UID: "3f3f800d-e685-42ae-b7a8-df2ede6cb623"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:48.839648 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.839614 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3f800d-e685-42ae-b7a8-df2ede6cb623-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:48.839648 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.839643 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3f800d-e685-42ae-b7a8-df2ede6cb623-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:48.839648 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:48.839655 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzqp\" (UniqueName: \"kubernetes.io/projected/3f3f800d-e685-42ae-b7a8-df2ede6cb623-kube-api-access-7dzqp\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:49.373178 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.373093 5117 generic.go:356] "Generic (PLEG): container finished" podID="3f3f800d-e685-42ae-b7a8-df2ede6cb623" containerID="316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07" exitCode=0 Nov 03 04:24:49.373178 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.373147 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" Nov 03 04:24:49.373178 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.373169 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" event={"ID":"3f3f800d-e685-42ae-b7a8-df2ede6cb623","Type":"ContainerDied","Data":"316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07"} Nov 03 04:24:49.373774 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.373214 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf" event={"ID":"3f3f800d-e685-42ae-b7a8-df2ede6cb623","Type":"ContainerDied","Data":"01cd09d17209e54a88dc5b99ae37b1a2726f96e7de43da7b1faa5937b2f2fbd3"} Nov 03 04:24:49.373774 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.373237 5117 scope.go:122] "RemoveContainer" containerID="316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07" Nov 03 04:24:49.382733 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.382706 5117 scope.go:122] "RemoveContainer" containerID="316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07" Nov 03 04:24:49.383073 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:24:49.383050 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07\": container with ID starting with 316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07 not found: ID does not exist" containerID="316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07" Nov 03 04:24:49.383129 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.383082 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07"} err="failed to get container status \"316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07\": rpc error: code = NotFound desc = could not find container \"316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07\": container with ID starting with 316066ccab27b77f98c81bd322ecc47c6ffe64614599a36764cbb2e24f9fcc07 not found: ID does not exist" Nov 03 04:24:49.541231 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.541183 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf"] Nov 03 04:24:49.590521 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.590473 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-66d902/rate-test-model-66d902-kserve-f6f9fd9df-876pf"] Nov 03 04:24:49.883223 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.883180 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" podUID="6735ef43-0f84-4832-b8be-6947a0401721" containerName="main" containerID="cri-o://b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736" gracePeriod=45 Nov 03 04:24:49.932671 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:49.932607 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" podUID="38375563-5dc0-4101-93af-ad9978d8997b" containerName="main" containerID="cri-o://db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b" gracePeriod=45 Nov 03 04:24:50.072106 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.072060 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" podUID="6735ef43-0f84-4832-b8be-6947a0401721" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": dial tcp 10.132.0.85:8000: connect: connection refused" Nov 03 04:24:50.199339 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.199311 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 04:24:50.230392 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.230363 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 04:24:50.356020 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.355983 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-tmp-dir\") pod \"6735ef43-0f84-4832-b8be-6947a0401721\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " Nov 03 04:24:50.356282 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356043 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-home\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-home\") pod \"38375563-5dc0-4101-93af-ad9978d8997b\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " Nov 03 04:24:50.356282 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356126 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38375563-5dc0-4101-93af-ad9978d8997b-tls-certs\" (UniqueName: \"kubernetes.io/secret/38375563-5dc0-4101-93af-ad9978d8997b-tls-certs\") pod \"38375563-5dc0-4101-93af-ad9978d8997b\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " Nov 03 04:24:50.356282 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356158 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-tmp-dir\") pod \"38375563-5dc0-4101-93af-ad9978d8997b\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " Nov 03 04:24:50.356282 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356186 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6735ef43-0f84-4832-b8be-6947a0401721-kube-api-access-8lpzm\" (UniqueName: \"kubernetes.io/projected/6735ef43-0f84-4832-b8be-6947a0401721-kube-api-access-8lpzm\") pod \"6735ef43-0f84-4832-b8be-6947a0401721\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " Nov 03 04:24:50.356282 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356223 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38375563-5dc0-4101-93af-ad9978d8997b-kube-api-access-kqpqd\" (UniqueName: \"kubernetes.io/projected/38375563-5dc0-4101-93af-ad9978d8997b-kube-api-access-kqpqd\") pod \"38375563-5dc0-4101-93af-ad9978d8997b\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " Nov 03 04:24:50.356282 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356250 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-dshm\") pod \"6735ef43-0f84-4832-b8be-6947a0401721\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " Nov 03 04:24:50.356640 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356307 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-dshm\") pod \"38375563-5dc0-4101-93af-ad9978d8997b\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " Nov 03 04:24:50.356640 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356304 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-tmp-dir" pod "6735ef43-0f84-4832-b8be-6947a0401721" (UID: "6735ef43-0f84-4832-b8be-6947a0401721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.356640 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356344 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6735ef43-0f84-4832-b8be-6947a0401721-tls-certs\" (UniqueName: \"kubernetes.io/secret/6735ef43-0f84-4832-b8be-6947a0401721-tls-certs\") pod \"6735ef43-0f84-4832-b8be-6947a0401721\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " Nov 03 04:24:50.356640 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356392 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-model-cache\") pod \"38375563-5dc0-4101-93af-ad9978d8997b\" (UID: \"38375563-5dc0-4101-93af-ad9978d8997b\") " Nov 03 04:24:50.356640 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356416 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-model-cache\") pod \"6735ef43-0f84-4832-b8be-6947a0401721\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " Nov 03 04:24:50.356640 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356463 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-home\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-home\") pod \"6735ef43-0f84-4832-b8be-6947a0401721\" (UID: \"6735ef43-0f84-4832-b8be-6947a0401721\") " Nov 03 04:24:50.356640 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356524 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-tmp-dir" pod "38375563-5dc0-4101-93af-ad9978d8997b" (UID: "38375563-5dc0-4101-93af-ad9978d8997b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.356972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356689 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-home" pod "38375563-5dc0-4101-93af-ad9978d8997b" (UID: "38375563-5dc0-4101-93af-ad9978d8997b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.356972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356894 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.356972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356911 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.356972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356916 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-home" pod "6735ef43-0f84-4832-b8be-6947a0401721" (UID: "6735ef43-0f84-4832-b8be-6947a0401721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.356972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.356924 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.357401 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.357368 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-model-cache" pod "38375563-5dc0-4101-93af-ad9978d8997b" (UID: "38375563-5dc0-4101-93af-ad9978d8997b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.359821 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.359791 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-model-cache" pod "6735ef43-0f84-4832-b8be-6947a0401721" (UID: "6735ef43-0f84-4832-b8be-6947a0401721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.360261 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.360051 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38375563-5dc0-4101-93af-ad9978d8997b-tls-certs" pod "38375563-5dc0-4101-93af-ad9978d8997b" (UID: "38375563-5dc0-4101-93af-ad9978d8997b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:50.360508 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.360265 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-dshm" pod "38375563-5dc0-4101-93af-ad9978d8997b" (UID: "38375563-5dc0-4101-93af-ad9978d8997b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.360508 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.360400 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38375563-5dc0-4101-93af-ad9978d8997b-kube-api-access-kqpqd" pod "38375563-5dc0-4101-93af-ad9978d8997b" (UID: "38375563-5dc0-4101-93af-ad9978d8997b"). InnerVolumeSpecName "kube-api-access-kqpqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:50.360508 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.360424 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6735ef43-0f84-4832-b8be-6947a0401721-kube-api-access-8lpzm" pod "6735ef43-0f84-4832-b8be-6947a0401721" (UID: "6735ef43-0f84-4832-b8be-6947a0401721"). InnerVolumeSpecName "kube-api-access-8lpzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:24:50.360845 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.360827 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-dshm" pod "6735ef43-0f84-4832-b8be-6947a0401721" (UID: "6735ef43-0f84-4832-b8be-6947a0401721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:24:50.361764 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.361745 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6735ef43-0f84-4832-b8be-6947a0401721-tls-certs" pod "6735ef43-0f84-4832-b8be-6947a0401721" (UID: "6735ef43-0f84-4832-b8be-6947a0401721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:24:50.382295 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.382256 5117 generic.go:356] "Generic (PLEG): container finished" podID="6735ef43-0f84-4832-b8be-6947a0401721" containerID="b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736" exitCode=0 Nov 03 04:24:50.382865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.382358 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" Nov 03 04:24:50.382865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.382377 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" event={"ID":"6735ef43-0f84-4832-b8be-6947a0401721","Type":"ContainerDied","Data":"b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736"} Nov 03 04:24:50.382865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.382420 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p" event={"ID":"6735ef43-0f84-4832-b8be-6947a0401721","Type":"ContainerDied","Data":"e30aeb7cc0ad7d55de7f235f33cd1e9e05fc69a494e837f157b46aacecb54f90"} Nov 03 04:24:50.382865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.382472 5117 scope.go:122] "RemoveContainer" containerID="b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736" Nov 03 04:24:50.384361 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.384335 5117 generic.go:356] "Generic (PLEG): container finished" podID="38375563-5dc0-4101-93af-ad9978d8997b" containerID="db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b" exitCode=0 Nov 03 04:24:50.384519 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.384468 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" Nov 03 04:24:50.384620 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.384599 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" event={"ID":"38375563-5dc0-4101-93af-ad9978d8997b","Type":"ContainerDied","Data":"db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b"} Nov 03 04:24:50.384692 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.384642 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" event={"ID":"38375563-5dc0-4101-93af-ad9978d8997b","Type":"ContainerDied","Data":"d204316c2c6e4682f8af51d41eed7fc51d1210e9475b808b116b2a73ab1c266b"} Nov 03 04:24:50.393308 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.393290 5117 scope.go:122] "RemoveContainer" containerID="b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736" Nov 03 04:24:50.393666 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:24:50.393638 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736\": container with ID starting with b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736 not found: ID does not exist" containerID="b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736" Nov 03 04:24:50.393773 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.393678 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736"} err="failed to get container status \"b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736\": rpc error: code = NotFound desc = could not find container \"b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736\": container with ID starting with b4d1c06cea7f4ad6b4128a16bf0c1510fc1a9a2a2b8f01c34b85c854d4042736 not found: ID does not exist" Nov 03 04:24:50.393773 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.393702 5117 scope.go:122] "RemoveContainer" containerID="db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b" Nov 03 04:24:50.403076 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.403028 5117 scope.go:122] "RemoveContainer" containerID="db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b" Nov 03 04:24:50.403358 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:24:50.403325 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b\": container with ID starting with db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b not found: ID does not exist" containerID="db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b" Nov 03 04:24:50.403472 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.403367 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b"} err="failed to get container status \"db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b\": rpc error: code = NotFound desc = could not find container \"db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b\": container with ID starting with db32fa54cc2bbb0a51fa48f8063ab8d026c8c5623f2860671aa9d50ed0fdf41b not found: ID does not exist" Nov 03 04:24:50.457714 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457674 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457714 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457714 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457731 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457745 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38375563-5dc0-4101-93af-ad9978d8997b-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457758 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpzm\" (UniqueName: \"kubernetes.io/projected/6735ef43-0f84-4832-b8be-6947a0401721-kube-api-access-8lpzm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457774 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpqd\" (UniqueName: \"kubernetes.io/projected/38375563-5dc0-4101-93af-ad9978d8997b-kube-api-access-kqpqd\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457788 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6735ef43-0f84-4832-b8be-6947a0401721-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457800 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38375563-5dc0-4101-93af-ad9978d8997b-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.457934 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.457813 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6735ef43-0f84-4832-b8be-6947a0401721-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:24:50.559216 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.559178 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p"] Nov 03 04:24:50.563669 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.563640 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-2-w0-0aa6b9-kserve-588bdcb7cf-swv4p"] Nov 03 04:24:50.672311 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.672225 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh"] Nov 03 04:24:50.674181 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:50.674157 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh"] Nov 03 04:24:51.088886 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:51.088845 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38375563-5dc0-4101-93af-ad9978d8997b" path="/var/lib/kubelet/pods/38375563-5dc0-4101-93af-ad9978d8997b/volumes" Nov 03 04:24:51.089264 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:51.089250 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3f800d-e685-42ae-b7a8-df2ede6cb623" path="/var/lib/kubelet/pods/3f3f800d-e685-42ae-b7a8-df2ede6cb623/volumes" Nov 03 04:24:51.089628 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:51.089615 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6735ef43-0f84-4832-b8be-6947a0401721" path="/var/lib/kubelet/pods/6735ef43-0f84-4832-b8be-6947a0401721/volumes" Nov 03 04:24:51.209921 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:24:51.209870 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/distinct-w0-0aa6b9-kserve-6f769fdb76-gdtgh" podUID="38375563-5dc0-4101-93af-ad9978d8997b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 03 04:25:00.011191 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:25:00.011140 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:25:11.295190 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.295153 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9"] Nov 03 04:25:11.295724 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.295387 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" podUID="1c357377-c079-42a9-88d5-ecdf1e32504b" containerName="maas-api" containerID="cri-o://083d4db5eb49845aa21211bea69f19f349f7dff720ee51465cfac4371a4f8b2f" gracePeriod=30 Nov 03 04:25:11.510730 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.510681 5117 generic.go:356] "Generic (PLEG): container finished" podID="1c357377-c079-42a9-88d5-ecdf1e32504b" containerID="083d4db5eb49845aa21211bea69f19f349f7dff720ee51465cfac4371a4f8b2f" exitCode=0 Nov 03 04:25:11.510952 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.510763 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" event={"ID":"1c357377-c079-42a9-88d5-ecdf1e32504b","Type":"ContainerDied","Data":"083d4db5eb49845aa21211bea69f19f349f7dff720ee51465cfac4371a4f8b2f"} Nov 03 04:25:11.572358 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.572330 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 04:25:11.664562 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.664524 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c357377-c079-42a9-88d5-ecdf1e32504b-kube-api-access-72lbs\" (UniqueName: \"kubernetes.io/projected/1c357377-c079-42a9-88d5-ecdf1e32504b-kube-api-access-72lbs\") pod \"1c357377-c079-42a9-88d5-ecdf1e32504b\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " Nov 03 04:25:11.664745 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.664574 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls\") pod \"1c357377-c079-42a9-88d5-ecdf1e32504b\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " Nov 03 04:25:11.664745 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.664605 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-metrics-certs\") pod \"1c357377-c079-42a9-88d5-ecdf1e32504b\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " Nov 03 04:25:11.664835 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.664785 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c357377-c079-42a9-88d5-ecdf1e32504b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c357377-c079-42a9-88d5-ecdf1e32504b-openshift-service-ca\") pod \"1c357377-c079-42a9-88d5-ecdf1e32504b\" (UID: \"1c357377-c079-42a9-88d5-ecdf1e32504b\") " Nov 03 04:25:11.665222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.665190 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c357377-c079-42a9-88d5-ecdf1e32504b-openshift-service-ca" pod "1c357377-c079-42a9-88d5-ecdf1e32504b" (UID: "1c357377-c079-42a9-88d5-ecdf1e32504b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:25:11.667022 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.666991 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls" pod "1c357377-c079-42a9-88d5-ecdf1e32504b" (UID: "1c357377-c079-42a9-88d5-ecdf1e32504b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:11.667157 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.666995 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c357377-c079-42a9-88d5-ecdf1e32504b-kube-api-access-72lbs" pod "1c357377-c079-42a9-88d5-ecdf1e32504b" (UID: "1c357377-c079-42a9-88d5-ecdf1e32504b"). InnerVolumeSpecName "kube-api-access-72lbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:11.667236 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.667212 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-metrics-certs" pod "1c357377-c079-42a9-88d5-ecdf1e32504b" (UID: "1c357377-c079-42a9-88d5-ecdf1e32504b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:11.766564 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.766522 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c357377-c079-42a9-88d5-ecdf1e32504b-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:11.766564 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.766559 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72lbs\" (UniqueName: \"kubernetes.io/projected/1c357377-c079-42a9-88d5-ecdf1e32504b-kube-api-access-72lbs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:11.766564 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.766572 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:11.766794 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:11.766582 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c357377-c079-42a9-88d5-ecdf1e32504b-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:12.521618 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:12.521580 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" event={"ID":"1c357377-c079-42a9-88d5-ecdf1e32504b","Type":"ContainerDied","Data":"2aa0420481960d6a8bb5c087cb0ec2a66ad5b71a99883022796b8341eb833b55"} Nov 03 04:25:12.522137 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:12.521674 5117 scope.go:122] "RemoveContainer" containerID="083d4db5eb49845aa21211bea69f19f349f7dff720ee51465cfac4371a4f8b2f" Nov 03 04:25:12.522137 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:12.521826 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9" Nov 03 04:25:12.690359 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:12.690285 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9"] Nov 03 04:25:12.740460 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:12.740409 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-66d902-75bbc7698d-8z2v9"] Nov 03 04:25:13.090164 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:13.090132 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c357377-c079-42a9-88d5-ecdf1e32504b" path="/var/lib/kubelet/pods/1c357377-c079-42a9-88d5-ecdf1e32504b/volumes" Nov 03 04:25:30.011289 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:25:30.011233 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:25:34.648093 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:34.648058 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr"] Nov 03 04:25:34.648711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:34.648251 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" podUID="99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" containerName="maas-api" containerID="cri-o://dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d" gracePeriod=30 Nov 03 04:25:34.914215 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:34.914153 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 04:25:35.093054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.093022 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-openshift-service-ca\") pod \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " Nov 03 04:25:35.093240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.093099 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-maas-api-tls\") pod \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " Nov 03 04:25:35.093240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.093119 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-kube-api-access-hzlvh\" (UniqueName: \"kubernetes.io/projected/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-kube-api-access-hzlvh\") pod \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " Nov 03 04:25:35.093240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.093171 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-metrics-certs\") pod \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\" (UID: \"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3\") " Nov 03 04:25:35.093435 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.093353 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-openshift-service-ca" pod "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" (UID: "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:25:35.093532 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.093514 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:35.095406 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.095380 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-kube-api-access-hzlvh" pod "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" (UID: "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3"). InnerVolumeSpecName "kube-api-access-hzlvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:35.095534 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.095436 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-maas-api-tls" pod "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" (UID: "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:35.095534 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.095467 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-metrics-certs" pod "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" (UID: "99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:35.194220 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.194116 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:35.194220 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.194155 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzlvh\" (UniqueName: \"kubernetes.io/projected/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-kube-api-access-hzlvh\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:35.194220 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.194170 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:35.653339 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.653303 5117 generic.go:356] "Generic (PLEG): container finished" podID="99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" containerID="dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d" exitCode=0 Nov 03 04:25:35.653865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.653346 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" event={"ID":"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3","Type":"ContainerDied","Data":"dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d"} Nov 03 04:25:35.653865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.653359 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" Nov 03 04:25:35.653865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.653387 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr" event={"ID":"99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3","Type":"ContainerDied","Data":"4bbbe212edd81fe5be8ffcbe3b5318971ef8c49b2a5efe8d4d9db1049ffad258"} Nov 03 04:25:35.653865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.653409 5117 scope.go:122] "RemoveContainer" containerID="dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d" Nov 03 04:25:35.663417 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.663401 5117 scope.go:122] "RemoveContainer" containerID="dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d" Nov 03 04:25:35.663740 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:25:35.663717 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d\": container with ID starting with dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d not found: ID does not exist" containerID="dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d" Nov 03 04:25:35.663796 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.663751 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d"} err="failed to get container status \"dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d\": rpc error: code = NotFound desc = could not find container \"dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d\": container with ID starting with dc1fa8d96a1b6468d65da23dd9cedd4bc6476a3a10c303e7579d3b5ff6a8506d not found: ID does not exist" Nov 03 04:25:35.820008 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.819977 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr"] Nov 03 04:25:35.870614 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:35.870567 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3d8dc5-85668554d7-gv2kr"] Nov 03 04:25:37.089757 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:37.089717 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" path="/var/lib/kubelet/pods/99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3/volumes" Nov 03 04:25:44.137786 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:44.137749 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd"] Nov 03 04:25:44.171120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:44.167985 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf"] Nov 03 04:25:59.158528 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.158467 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" containerID="cri-o://86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355" gracePeriod=45 Nov 03 04:25:59.189089 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.189049 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" containerID="cri-o://25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb" gracePeriod=45 Nov 03 04:25:59.337477 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.337416 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Nov 03 04:25:59.462714 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.462678 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 04:25:59.487972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.487940 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 04:25:59.527771 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.527735 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-model-cache\") pod \"f23474ee-259b-4255-9c90-0197447f3c61\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " Nov 03 04:25:59.527972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.527778 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23474ee-259b-4255-9c90-0197447f3c61-tls-certs\" (UniqueName: \"kubernetes.io/secret/f23474ee-259b-4255-9c90-0197447f3c61-tls-certs\") pod \"f23474ee-259b-4255-9c90-0197447f3c61\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " Nov 03 04:25:59.527972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.527813 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23474ee-259b-4255-9c90-0197447f3c61-kube-api-access-fkn2s\" (UniqueName: \"kubernetes.io/projected/f23474ee-259b-4255-9c90-0197447f3c61-kube-api-access-fkn2s\") pod \"f23474ee-259b-4255-9c90-0197447f3c61\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " Nov 03 04:25:59.527972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.527872 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-home\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-home\") pod \"f23474ee-259b-4255-9c90-0197447f3c61\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " Nov 03 04:25:59.527972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.527930 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-tmp-dir\") pod \"f23474ee-259b-4255-9c90-0197447f3c61\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " Nov 03 04:25:59.528197 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.527991 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-dshm\") pod \"f23474ee-259b-4255-9c90-0197447f3c61\" (UID: \"f23474ee-259b-4255-9c90-0197447f3c61\") " Nov 03 04:25:59.528197 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.528023 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-model-cache" pod "f23474ee-259b-4255-9c90-0197447f3c61" (UID: "f23474ee-259b-4255-9c90-0197447f3c61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.528297 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.528235 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-home" pod "f23474ee-259b-4255-9c90-0197447f3c61" (UID: "f23474ee-259b-4255-9c90-0197447f3c61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.528297 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.528277 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-tmp-dir" pod "f23474ee-259b-4255-9c90-0197447f3c61" (UID: "f23474ee-259b-4255-9c90-0197447f3c61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.528387 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.528307 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.528387 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.528325 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.530373 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.530352 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-dshm" pod "f23474ee-259b-4255-9c90-0197447f3c61" (UID: "f23474ee-259b-4255-9c90-0197447f3c61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.530723 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.530700 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23474ee-259b-4255-9c90-0197447f3c61-tls-certs" pod "f23474ee-259b-4255-9c90-0197447f3c61" (UID: "f23474ee-259b-4255-9c90-0197447f3c61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:59.530723 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.530716 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23474ee-259b-4255-9c90-0197447f3c61-kube-api-access-fkn2s" pod "f23474ee-259b-4255-9c90-0197447f3c61" (UID: "f23474ee-259b-4255-9c90-0197447f3c61"). InnerVolumeSpecName "kube-api-access-fkn2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:59.629403 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629360 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-dshm\") pod \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " Nov 03 04:25:59.629623 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629423 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-home\") pod \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " Nov 03 04:25:59.629623 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629525 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-model-cache\") pod \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " Nov 03 04:25:59.629623 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629583 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-kube-api-access-8nnm8\" (UniqueName: \"kubernetes.io/projected/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-kube-api-access-8nnm8\") pod \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " Nov 03 04:25:59.629823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629635 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tmp-dir\") pod \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " Nov 03 04:25:59.629823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629668 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tls-certs\") pod \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\" (UID: \"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875\") " Nov 03 04:25:59.629823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629722 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-home" pod "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" (UID: "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.629823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.629756 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-model-cache" pod "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" (UID: "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.630130 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630106 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tmp-dir" pod "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" (UID: "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.630262 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630242 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.630262 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630264 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23474ee-259b-4255-9c90-0197447f3c61-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.630652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630279 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkn2s\" (UniqueName: \"kubernetes.io/projected/f23474ee-259b-4255-9c90-0197447f3c61-kube-api-access-fkn2s\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.630652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630296 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.630652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630314 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.630652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630328 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.630652 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.630340 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23474ee-259b-4255-9c90-0197447f3c61-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.632325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.632268 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-kube-api-access-8nnm8" pod "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" (UID: "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875"). InnerVolumeSpecName "kube-api-access-8nnm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:25:59.632634 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.632602 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-dshm" pod "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" (UID: "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:25:59.632957 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.632933 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tls-certs" pod "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" (UID: "a6eb7f13-1d86-4d51-bc3b-25f3b10fe875"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:25:59.731493 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.731379 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.731493 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.731411 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nnm8\" (UniqueName: \"kubernetes.io/projected/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-kube-api-access-8nnm8\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.731493 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.731421 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:25:59.782199 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.782162 5117 generic.go:356] "Generic (PLEG): container finished" podID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerID="25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb" exitCode=0 Nov 03 04:25:59.782407 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.782242 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" Nov 03 04:25:59.782407 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.782252 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" event={"ID":"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875","Type":"ContainerDied","Data":"25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb"} Nov 03 04:25:59.782407 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.782300 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf" event={"ID":"a6eb7f13-1d86-4d51-bc3b-25f3b10fe875","Type":"ContainerDied","Data":"3f4706a0cbd211a424a98ad411c496a704a97ef3fbacddeb63c83c44fc13887d"} Nov 03 04:25:59.782407 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.782323 5117 scope.go:122] "RemoveContainer" containerID="25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb" Nov 03 04:25:59.785419 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.785382 5117 generic.go:356] "Generic (PLEG): container finished" podID="f23474ee-259b-4255-9c90-0197447f3c61" containerID="86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355" exitCode=0 Nov 03 04:25:59.785583 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.785478 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" Nov 03 04:25:59.785643 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.785597 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" event={"ID":"f23474ee-259b-4255-9c90-0197447f3c61","Type":"ContainerDied","Data":"86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355"} Nov 03 04:25:59.785643 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.785635 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd" event={"ID":"f23474ee-259b-4255-9c90-0197447f3c61","Type":"ContainerDied","Data":"24195fc357d4d9dceb4019b7cd6453cf5ead0b169a036e32fb0d352b7efaf6f8"} Nov 03 04:25:59.795009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.794892 5117 scope.go:122] "RemoveContainer" containerID="25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb" Nov 03 04:25:59.795277 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:25:59.795255 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb\": container with ID starting with 25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb not found: ID does not exist" containerID="25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb" Nov 03 04:25:59.795352 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.795290 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb"} err="failed to get container status \"25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb\": rpc error: code = NotFound desc = could not find container \"25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb\": container with ID starting with 25854de71af20161a74e5cc444786e892e6da60c8503cee2c7d26702c707ddeb not found: ID does not exist" Nov 03 04:25:59.795352 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.795316 5117 scope.go:122] "RemoveContainer" containerID="86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355" Nov 03 04:25:59.813376 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.813348 5117 scope.go:122] "RemoveContainer" containerID="86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355" Nov 03 04:25:59.813791 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:25:59.813765 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355\": container with ID starting with 86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355 not found: ID does not exist" containerID="86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355" Nov 03 04:25:59.813899 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.813800 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355"} err="failed to get container status \"86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355\": rpc error: code = NotFound desc = could not find container \"86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355\": container with ID starting with 86b958f5d9eef7094da2ffad1f9860efce42fe04cabc788a00e0f412f1cb3355 not found: ID does not exist" Nov 03 04:25:59.950235 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.950199 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf"] Nov 03 04:25:59.954296 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:25:59.954266 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-w2-de27f5-kserve-56b4df96bb-ddlrf"] Nov 03 04:26:00.011086 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:26:00.010989 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:26:00.065697 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:00.065663 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd"] Nov 03 04:26:00.070321 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:00.070293 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-de27f5/distinct-2-w2-de27f5-kserve-7f4d6cbf-ftqmd"] Nov 03 04:26:01.089123 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:01.089090 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" path="/var/lib/kubelet/pods/a6eb7f13-1d86-4d51-bc3b-25f3b10fe875/volumes" Nov 03 04:26:01.089549 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:01.089533 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23474ee-259b-4255-9c90-0197447f3c61" path="/var/lib/kubelet/pods/f23474ee-259b-4255-9c90-0197447f3c61/volumes" Nov 03 04:26:07.880803 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:07.880703 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:26:07.902999 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:07.886569 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:26:07.902999 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:07.898733 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:26:07.904350 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:07.904329 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:26:11.301765 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.301730 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s"] Nov 03 04:26:11.302186 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.301939 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" podUID="569c995c-b622-474e-ad2b-1c70fa3a6d86" containerName="maas-api" containerID="cri-o://592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf" gracePeriod=30 Nov 03 04:26:11.592580 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.592551 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 04:26:11.755120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.755087 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569c995c-b622-474e-ad2b-1c70fa3a6d86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c995c-b622-474e-ad2b-1c70fa3a6d86-openshift-service-ca\") pod \"569c995c-b622-474e-ad2b-1c70fa3a6d86\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " Nov 03 04:26:11.755287 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.755128 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569c995c-b622-474e-ad2b-1c70fa3a6d86-kube-api-access-79hd2\" (UniqueName: \"kubernetes.io/projected/569c995c-b622-474e-ad2b-1c70fa3a6d86-kube-api-access-79hd2\") pod \"569c995c-b622-474e-ad2b-1c70fa3a6d86\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " Nov 03 04:26:11.755287 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.755234 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls\") pod \"569c995c-b622-474e-ad2b-1c70fa3a6d86\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " Nov 03 04:26:11.755287 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.755256 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-metrics-certs\") pod \"569c995c-b622-474e-ad2b-1c70fa3a6d86\" (UID: \"569c995c-b622-474e-ad2b-1c70fa3a6d86\") " Nov 03 04:26:11.755467 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.755429 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c995c-b622-474e-ad2b-1c70fa3a6d86-openshift-service-ca" pod "569c995c-b622-474e-ad2b-1c70fa3a6d86" (UID: "569c995c-b622-474e-ad2b-1c70fa3a6d86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:26:11.757538 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.757507 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls" pod "569c995c-b622-474e-ad2b-1c70fa3a6d86" (UID: "569c995c-b622-474e-ad2b-1c70fa3a6d86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:26:11.757636 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.757537 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-metrics-certs" pod "569c995c-b622-474e-ad2b-1c70fa3a6d86" (UID: "569c995c-b622-474e-ad2b-1c70fa3a6d86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:26:11.757636 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.757550 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c995c-b622-474e-ad2b-1c70fa3a6d86-kube-api-access-79hd2" pod "569c995c-b622-474e-ad2b-1c70fa3a6d86" (UID: "569c995c-b622-474e-ad2b-1c70fa3a6d86"). InnerVolumeSpecName "kube-api-access-79hd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:26:11.852040 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.851948 5117 generic.go:356] "Generic (PLEG): container finished" podID="569c995c-b622-474e-ad2b-1c70fa3a6d86" containerID="592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf" exitCode=0 Nov 03 04:26:11.852040 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.852000 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" Nov 03 04:26:11.852248 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.852037 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" event={"ID":"569c995c-b622-474e-ad2b-1c70fa3a6d86","Type":"ContainerDied","Data":"592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf"} Nov 03 04:26:11.852248 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.852129 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s" event={"ID":"569c995c-b622-474e-ad2b-1c70fa3a6d86","Type":"ContainerDied","Data":"c170bc0129a8f0599f037119812b4925ae53907f7d1a3b2c69d1fd48c628faa1"} Nov 03 04:26:11.852248 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.852159 5117 scope.go:122] "RemoveContainer" containerID="592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf" Nov 03 04:26:11.856863 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.856830 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:11.857005 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.856865 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569c995c-b622-474e-ad2b-1c70fa3a6d86-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:11.857005 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.856882 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569c995c-b622-474e-ad2b-1c70fa3a6d86-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:11.857005 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.856898 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79hd2\" (UniqueName: \"kubernetes.io/projected/569c995c-b622-474e-ad2b-1c70fa3a6d86-kube-api-access-79hd2\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:26:11.862360 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.862340 5117 scope.go:122] "RemoveContainer" containerID="592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf" Nov 03 04:26:11.862714 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:26:11.862692 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf\": container with ID starting with 592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf not found: ID does not exist" containerID="592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf" Nov 03 04:26:11.862788 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:11.862723 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf"} err="failed to get container status \"592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf\": rpc error: code = NotFound desc = could not find container \"592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf\": container with ID starting with 592e0f148eafa581b0d36b244e36b73b6ebea6bbbdeff78100f42833d528addf not found: ID does not exist" Nov 03 04:26:12.019974 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:12.019939 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s"] Nov 03 04:26:12.023453 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:12.023417 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-de27f5-6bb74f774f-62g7s"] Nov 03 04:26:13.088697 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:13.088664 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569c995c-b622-474e-ad2b-1c70fa3a6d86" path="/var/lib/kubelet/pods/569c995c-b622-474e-ad2b-1c70fa3a6d86/volumes" Nov 03 04:26:28.657643 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:28.657609 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9szxj_70c7e0ef-af42-493c-baf8-1322dcbd8074/manager/0.log" Nov 03 04:26:30.011389 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:26:30.011346 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:26:32.053122 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:32.053086 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jnblx_0613f917-8742-408c-80ed-640e913c05f6/manager/0.log" Nov 03 04:26:33.091291 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:33.091243 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7957588c98-fsqfj_4c3696fd-b5a8-4c37-ad9b-888a384beb12/manager/1.log" Nov 03 04:26:34.082411 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:34.082372 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mn4ss_636bbe26-5388-49da-93d3-7d53a66b7d94/manager/0.log" Nov 03 04:26:34.344139 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:34.344051 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pzvwf_9699142d-1bf4-46a4-9705-f5bd4c028ae6/proxy/0.log" Nov 03 04:26:35.042376 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:26:35.042338 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7957588c98-fsqfj_4c3696fd-b5a8-4c37-ad9b-888a384beb12/manager/0.log" Nov 03 04:27:00.012018 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:27:00.011970 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:27:30.012404 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:27:30.012360 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:27:50.539061 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.539018 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7957588c98-fsqfj"] Nov 03 04:27:50.539751 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.539312 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7957588c98-fsqfj" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerName="manager" containerID="cri-o://fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305" gracePeriod=30 Nov 03 04:27:50.800734 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.800676 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 04:27:50.974840 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.974801 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-webhook-cert\") pod \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " Nov 03 04:27:50.975034 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.974870 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3696fd-b5a8-4c37-ad9b-888a384beb12-kube-api-access-4pzhm\" (UniqueName: \"kubernetes.io/projected/4c3696fd-b5a8-4c37-ad9b-888a384beb12-kube-api-access-4pzhm\") pod \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " Nov 03 04:27:50.975034 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.974922 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-metrics-certs\") pod \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\" (UID: \"4c3696fd-b5a8-4c37-ad9b-888a384beb12\") " Nov 03 04:27:50.977094 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.977061 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-metrics-certs" pod "4c3696fd-b5a8-4c37-ad9b-888a384beb12" (UID: "4c3696fd-b5a8-4c37-ad9b-888a384beb12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:27:50.977203 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.977128 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-webhook-cert" pod "4c3696fd-b5a8-4c37-ad9b-888a384beb12" (UID: "4c3696fd-b5a8-4c37-ad9b-888a384beb12"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:27:50.977203 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:50.977144 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3696fd-b5a8-4c37-ad9b-888a384beb12-kube-api-access-4pzhm" pod "4c3696fd-b5a8-4c37-ad9b-888a384beb12" (UID: "4c3696fd-b5a8-4c37-ad9b-888a384beb12"). InnerVolumeSpecName "kube-api-access-4pzhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:27:51.076539 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.076505 5117 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-webhook-cert\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:51.076539 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.076538 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzhm\" (UniqueName: \"kubernetes.io/projected/4c3696fd-b5a8-4c37-ad9b-888a384beb12-kube-api-access-4pzhm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:51.076750 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.076548 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3696fd-b5a8-4c37-ad9b-888a384beb12-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:27:51.385511 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.385418 5117 generic.go:356] "Generic (PLEG): container finished" podID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerID="fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305" exitCode=0 Nov 03 04:27:51.385511 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.385504 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-fsqfj" Nov 03 04:27:51.385724 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.385520 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-fsqfj" event={"ID":"4c3696fd-b5a8-4c37-ad9b-888a384beb12","Type":"ContainerDied","Data":"fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305"} Nov 03 04:27:51.385724 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.385582 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-fsqfj" event={"ID":"4c3696fd-b5a8-4c37-ad9b-888a384beb12","Type":"ContainerDied","Data":"58e9d9ed4171807a5ae67060df22f7901b61921bcda418d93c41b5345562df7d"} Nov 03 04:27:51.385724 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.385606 5117 scope.go:122] "RemoveContainer" containerID="fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305" Nov 03 04:27:51.400061 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.398917 5117 scope.go:122] "RemoveContainer" containerID="306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14" Nov 03 04:27:51.409603 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.409582 5117 scope.go:122] "RemoveContainer" containerID="fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305" Nov 03 04:27:51.409932 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:27:51.409911 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305\": container with ID starting with fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305 not found: ID does not exist" containerID="fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305" Nov 03 04:27:51.409991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.409942 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305"} err="failed to get container status \"fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305\": rpc error: code = NotFound desc = could not find container \"fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305\": container with ID starting with fcd694d5361504341cba46918594674e60d48e920759dd2f2f6c56c0d23ad305 not found: ID does not exist" Nov 03 04:27:51.409991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.409963 5117 scope.go:122] "RemoveContainer" containerID="306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14" Nov 03 04:27:51.410206 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:27:51.410189 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14\": container with ID starting with 306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14 not found: ID does not exist" containerID="306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14" Nov 03 04:27:51.410253 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.410211 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14"} err="failed to get container status \"306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14\": rpc error: code = NotFound desc = could not find container \"306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14\": container with ID starting with 306ab090749e3fd9af81d5e50c44a84b9250c89e8e950d951831726217f14d14 not found: ID does not exist" Nov 03 04:27:51.554299 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.554267 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7957588c98-fsqfj"] Nov 03 04:27:51.605234 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:51.605203 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7957588c98-fsqfj"] Nov 03 04:27:53.088707 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:27:53.088670 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" path="/var/lib/kubelet/pods/4c3696fd-b5a8-4c37-ad9b-888a384beb12/volumes" Nov 03 04:28:00.012328 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:28:00.012288 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:28:30.012967 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:28:30.012928 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:29:00.011145 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:29:00.011105 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:29:08.045381 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045341 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l"] Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045782 5117 cpu_manager.go:414] "containerMap: removing container" podUID="c8070701-d6c6-4399-88d0-298b5fca3efb" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045794 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8070701-d6c6-4399-88d0-298b5fca3efb" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045811 5117 cpu_manager.go:414] "containerMap: removing container" podUID="3f3f800d-e685-42ae-b7a8-df2ede6cb623" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045816 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3f800d-e685-42ae-b7a8-df2ede6cb623" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045828 5117 cpu_manager.go:414] "containerMap: removing container" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045834 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045842 5117 cpu_manager.go:414] "containerMap: removing container" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045847 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045856 5117 cpu_manager.go:414] "containerMap: removing container" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045861 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045869 5117 cpu_manager.go:414] "containerMap: removing container" podUID="b4e5fc07-68de-42ef-8737-354f5fee1710" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045874 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e5fc07-68de-42ef-8737-354f5fee1710" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045881 5117 cpu_manager.go:414] "containerMap: removing container" podUID="5294bb78-bd86-477b-b45e-d211e6cfd02f" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045885 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5294bb78-bd86-477b-b45e-d211e6cfd02f" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045893 5117 cpu_manager.go:414] "containerMap: removing container" podUID="1c357377-c079-42a9-88d5-ecdf1e32504b" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045897 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c357377-c079-42a9-88d5-ecdf1e32504b" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045904 5117 cpu_manager.go:414] "containerMap: removing container" podUID="38375563-5dc0-4101-93af-ad9978d8997b" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045908 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38375563-5dc0-4101-93af-ad9978d8997b" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045914 5117 cpu_manager.go:414] "containerMap: removing container" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerName="manager" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045919 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerName="manager" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045926 5117 cpu_manager.go:414] "containerMap: removing container" podUID="392157ef-9f8e-48b6-bdcd-da14bce0f10d" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045930 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392157ef-9f8e-48b6-bdcd-da14bce0f10d" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045938 5117 cpu_manager.go:414] "containerMap: removing container" podUID="51f3a7f5-ca55-471e-b18c-1bd44a8c1198" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045942 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f3a7f5-ca55-471e-b18c-1bd44a8c1198" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045950 5117 cpu_manager.go:414] "containerMap: removing container" podUID="569c995c-b622-474e-ad2b-1c70fa3a6d86" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045954 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569c995c-b622-474e-ad2b-1c70fa3a6d86" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045962 5117 cpu_manager.go:414] "containerMap: removing container" podUID="47833055-fc67-4320-b300-05193b9dd528" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045966 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47833055-fc67-4320-b300-05193b9dd528" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045972 5117 cpu_manager.go:414] "containerMap: removing container" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045976 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045982 5117 cpu_manager.go:414] "containerMap: removing container" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045987 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045993 5117 cpu_manager.go:414] "containerMap: removing container" podUID="6b43ac6c-1194-4622-b271-a669efeb6588" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.045997 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b43ac6c-1194-4622-b271-a669efeb6588" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046008 5117 cpu_manager.go:414] "containerMap: removing container" podUID="d4d334d3-ed81-4de0-9422-b9f8e1d123ad" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046012 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d334d3-ed81-4de0-9422-b9f8e1d123ad" containerName="maas-api" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046019 5117 cpu_manager.go:414] "containerMap: removing container" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerName="manager" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046023 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerName="manager" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046029 5117 cpu_manager.go:414] "containerMap: removing container" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046033 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" Nov 03 04:29:08.046012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046040 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046044 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046055 5117 cpu_manager.go:414] "containerMap: removing container" podUID="99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046060 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046066 5117 cpu_manager.go:414] "containerMap: removing container" podUID="6735ef43-0f84-4832-b8be-6947a0401721" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046070 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6735ef43-0f84-4832-b8be-6947a0401721" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046078 5117 cpu_manager.go:414] "containerMap: removing container" podUID="73a3d918-5836-4a33-b411-a05af4647b7b" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046082 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a3d918-5836-4a33-b411-a05af4647b7b" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046142 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="6735ef43-0f84-4832-b8be-6947a0401721" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046152 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a3d918-5836-4a33-b411-a05af4647b7b" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046158 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8070701-d6c6-4399-88d0-298b5fca3efb" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046164 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3f800d-e685-42ae-b7a8-df2ede6cb623" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046171 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2949fa5-406f-4816-af0a-c93db25f9db5" containerName="istio-proxy" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046176 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eb7f13-1d86-4d51-bc3b-25f3b10fe875" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046180 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d8f94f-9b7f-450e-8bbb-b42776fd7f94" containerName="istio-proxy" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046185 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e5fc07-68de-42ef-8737-354f5fee1710" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046190 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="5294bb78-bd86-477b-b45e-d211e6cfd02f" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046195 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c357377-c079-42a9-88d5-ecdf1e32504b" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046200 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="38375563-5dc0-4101-93af-ad9978d8997b" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046204 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerName="manager" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046210 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="392157ef-9f8e-48b6-bdcd-da14bce0f10d" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046215 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f3a7f5-ca55-471e-b18c-1bd44a8c1198" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046220 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="569c995c-b622-474e-ad2b-1c70fa3a6d86" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046226 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23474ee-259b-4255-9c90-0197447f3c61" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046230 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="47833055-fc67-4320-b300-05193b9dd528" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046235 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1aa63-6b48-40dd-ad68-ef76b84e6374" containerName="istio-proxy" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046240 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b43ac6c-1194-4622-b271-a669efeb6588" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046245 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d334d3-ed81-4de0-9422-b9f8e1d123ad" containerName="maas-api" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046250 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3696fd-b5a8-4c37-ad9b-888a384beb12" containerName="manager" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046254 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9c45e3-2dec-44ef-b877-0eadbf7c7e4f" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046259 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8333631b-54aa-4041-a8c6-a725077a336f" containerName="main" Nov 03 04:29:08.047346 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.046264 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bcaf2a-8d5b-4ff3-bff7-fed8c902c5a3" containerName="maas-api" Nov 03 04:29:08.049453 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.049428 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.082816 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.082771 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.082998 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.082833 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.082998 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.082945 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.082998 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.082972 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mxg9\" (UniqueName: \"kubernetes.io/projected/3bc1ec57-9eb0-4153-a266-cc11de2854a2-kube-api-access-6mxg9\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.083168 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.083151 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.083204 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.083194 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.083238 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.083214 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.083272 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.083238 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.083337 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.083325 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.163946 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.163902 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l"] Nov 03 04:29:08.184025 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.183978 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184243 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184081 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184243 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184122 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184243 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184184 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184243 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184216 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mxg9\" (UniqueName: \"kubernetes.io/projected/3bc1ec57-9eb0-4153-a266-cc11de2854a2-kube-api-access-6mxg9\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184480 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184297 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184480 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184325 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184480 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184351 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184687 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184658 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184827 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184689 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.184827 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184707 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.185033 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184841 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.185033 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.184743 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.185630 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.185604 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.186958 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.186934 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.187042 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.186963 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.247400 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.247362 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc1ec57-9eb0-4153-a266-cc11de2854a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.247628 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.247519 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mxg9\" (UniqueName: \"kubernetes.io/projected/3bc1ec57-9eb0-4153-a266-cc11de2854a2-kube-api-access-6mxg9\") pod \"maas-default-gateway-openshift-default-76f4876667-rnx7l\" (UID: \"3bc1ec57-9eb0-4153-a266-cc11de2854a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.362996 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.362960 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:08.791407 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.791378 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l"] Nov 03 04:29:08.792931 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:29:08.792901 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc1ec57_9eb0_4153_a266_cc11de2854a2.slice/crio-58f697aee9a2c78044137d45b3ca708beb829d1f1ef7f8233db66ef18796553e WatchSource:0}: Error finding container 58f697aee9a2c78044137d45b3ca708beb829d1f1ef7f8233db66ef18796553e: Status 404 returned error can't find the container with id 58f697aee9a2c78044137d45b3ca708beb829d1f1ef7f8233db66ef18796553e Nov 03 04:29:08.794997 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.794964 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:29:08.795082 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.795035 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:29:08.795082 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:08.795065 5117 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 04:29:09.784720 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:09.784681 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" event={"ID":"3bc1ec57-9eb0-4153-a266-cc11de2854a2","Type":"ContainerStarted","Data":"9a576b612bc486345c475e3ffe695505cd6b508bb2804a0f66efcf05ca540a01"} Nov 03 04:29:09.785132 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:09.784739 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" event={"ID":"3bc1ec57-9eb0-4153-a266-cc11de2854a2","Type":"ContainerStarted","Data":"58f697aee9a2c78044137d45b3ca708beb829d1f1ef7f8233db66ef18796553e"} Nov 03 04:29:09.960051 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:09.959993 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podStartSLOduration=1.959977019 podStartE2EDuration="1.959977019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:29:09.958611479 +0000 UTC m=+5284.989210018" watchObservedRunningTime="2026-11-03 04:29:09.959977019 +0000 UTC m=+5284.990575556" Nov 03 04:29:10.363740 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:10.363690 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:10.365358 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:10.365333 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 04:29:10.365431 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:10.365386 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 04:29:11.363546 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:11.363500 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 04:29:11.364067 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:11.363575 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 04:29:12.364548 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:12.364505 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 04:29:12.364991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:12.364580 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 04:29:13.363937 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:13.363893 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 04:29:13.364146 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:13.363964 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 04:29:14.364012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:14.363966 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 04:29:14.364438 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:14.364040 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 04:29:15.363901 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:15.363859 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 04:29:15.364065 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:15.363921 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 04:29:16.363663 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:16.363621 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 04:29:16.363856 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:16.363683 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 04:29:18.404702 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:18.404643 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:29:18.405258 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:18.404718 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:29:19.445658 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:19.445610 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:29:19.446176 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:19.445679 5117 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:29:19.453421 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:19.453385 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:19.453728 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:19.453699 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:19.454398 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:29:19.454378 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" Nov 03 04:29:30.018743 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:29:30.018695 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:30:00.010948 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:30:00.010897 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:30:00.171419 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.168603 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2"] Nov 03 04:30:00.174175 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.174140 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.229373 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.229334 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2"] Nov 03 04:30:00.277886 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.277794 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:30:00.285452 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.285403 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8d6x\" (UniqueName: \"kubernetes.io/projected/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-kube-api-access-b8d6x\") pod \"maas-api-key-cleanup-29894670-bhds2\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.285556 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.285533 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894670-bhds2\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.386466 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.386414 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894670-bhds2\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.386690 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.386543 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8d6x\" (UniqueName: \"kubernetes.io/projected/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-kube-api-access-b8d6x\") pod \"maas-api-key-cleanup-29894670-bhds2\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.387155 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.387126 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894670-bhds2\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.448945 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.448913 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8d6x\" (UniqueName: \"kubernetes.io/projected/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-kube-api-access-b8d6x\") pod \"maas-api-key-cleanup-29894670-bhds2\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.487930 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.487883 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:00.866336 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.866299 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2"] Nov 03 04:30:00.867885 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:30:00.867856 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a118a1a_7eb8_4032_aa7b_341b27d0fdb9.slice/crio-1112bec30bd86b23f65b810d95761de0b619af1f45f1710de3a29671943e8c9a WatchSource:0}: Error finding container 1112bec30bd86b23f65b810d95761de0b619af1f45f1710de3a29671943e8c9a: Status 404 returned error can't find the container with id 1112bec30bd86b23f65b810d95761de0b619af1f45f1710de3a29671943e8c9a Nov 03 04:30:00.869769 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:00.869750 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 04:30:01.116431 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:01.116345 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" event={"ID":"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9","Type":"ContainerStarted","Data":"1112bec30bd86b23f65b810d95761de0b619af1f45f1710de3a29671943e8c9a"} Nov 03 04:30:06.147469 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:06.147410 5117 generic.go:356] "Generic (PLEG): container finished" podID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerID="bba6af17075aedafb3d9785cf7e7f233d30aa586951a3eae3d74f5983263d017" exitCode=60 Nov 03 04:30:06.147928 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:06.147498 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" event={"ID":"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9","Type":"ContainerDied","Data":"bba6af17075aedafb3d9785cf7e7f233d30aa586951a3eae3d74f5983263d017"} Nov 03 04:30:06.147928 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:06.147782 5117 scope.go:122] "RemoveContainer" containerID="bba6af17075aedafb3d9785cf7e7f233d30aa586951a3eae3d74f5983263d017" Nov 03 04:30:07.155114 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:07.155078 5117 generic.go:356] "Generic (PLEG): container finished" podID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerID="5106f4f06bac31d8ee8c0b26247959622acd01eeb173b56a2c6af85c73b7af89" exitCode=60 Nov 03 04:30:07.155715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:07.155165 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" event={"ID":"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9","Type":"ContainerDied","Data":"5106f4f06bac31d8ee8c0b26247959622acd01eeb173b56a2c6af85c73b7af89"} Nov 03 04:30:07.155715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:07.155221 5117 scope.go:122] "RemoveContainer" containerID="bba6af17075aedafb3d9785cf7e7f233d30aa586951a3eae3d74f5983263d017" Nov 03 04:30:07.155715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:07.155507 5117 scope.go:122] "RemoveContainer" containerID="5106f4f06bac31d8ee8c0b26247959622acd01eeb173b56a2c6af85c73b7af89" Nov 03 04:30:07.155897 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:30:07.155827 5117 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894670-bhds2_odh-ai-gateway-infra(8a118a1a-7eb8-4032-aa7b-341b27d0fdb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" Nov 03 04:30:08.162242 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:08.162207 5117 scope.go:122] "RemoveContainer" containerID="5106f4f06bac31d8ee8c0b26247959622acd01eeb173b56a2c6af85c73b7af89" Nov 03 04:30:08.162745 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:30:08.162460 5117 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894670-bhds2_odh-ai-gateway-infra(8a118a1a-7eb8-4032-aa7b-341b27d0fdb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" Nov 03 04:30:21.083153 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:21.083118 5117 scope.go:122] "RemoveContainer" containerID="5106f4f06bac31d8ee8c0b26247959622acd01eeb173b56a2c6af85c73b7af89" Nov 03 04:30:22.233094 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:22.233018 5117 generic.go:356] "Generic (PLEG): container finished" podID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerID="80e505b32f849ddbddecd9ed7061779a2a1149897458406664a16b75c81e7533" exitCode=60 Nov 03 04:30:22.233548 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:22.233092 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" event={"ID":"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9","Type":"ContainerDied","Data":"80e505b32f849ddbddecd9ed7061779a2a1149897458406664a16b75c81e7533"} Nov 03 04:30:22.233548 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:22.233161 5117 scope.go:122] "RemoveContainer" containerID="5106f4f06bac31d8ee8c0b26247959622acd01eeb173b56a2c6af85c73b7af89" Nov 03 04:30:22.233684 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:22.233590 5117 scope.go:122] "RemoveContainer" containerID="80e505b32f849ddbddecd9ed7061779a2a1149897458406664a16b75c81e7533" Nov 03 04:30:22.233842 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:30:22.233824 5117 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894670-bhds2_odh-ai-gateway-infra(8a118a1a-7eb8-4032-aa7b-341b27d0fdb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" Nov 03 04:30:23.368264 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.368221 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2"] Nov 03 04:30:23.843472 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.818384 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:23.896411 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.896344 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-maas-api-service-ca\") pod \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " Nov 03 04:30:23.896991 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.896960 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-maas-api-service-ca" pod "8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" (UID: "8a118a1a-7eb8-4032-aa7b-341b27d0fdb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:30:23.897054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.897033 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-kube-api-access-b8d6x\" (UniqueName: \"kubernetes.io/projected/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-kube-api-access-b8d6x\") pod \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\" (UID: \"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9\") " Nov 03 04:30:23.897522 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.897496 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-maas-api-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:23.899038 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.899004 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-kube-api-access-b8d6x" pod "8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" (UID: "8a118a1a-7eb8-4032-aa7b-341b27d0fdb9"). InnerVolumeSpecName "kube-api-access-b8d6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:30:23.998286 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:23.998180 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8d6x\" (UniqueName: \"kubernetes.io/projected/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9-kube-api-access-b8d6x\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:30:24.270303 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:24.270214 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" event={"ID":"8a118a1a-7eb8-4032-aa7b-341b27d0fdb9","Type":"ContainerDied","Data":"1112bec30bd86b23f65b810d95761de0b619af1f45f1710de3a29671943e8c9a"} Nov 03 04:30:24.270543 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:24.270530 5117 scope.go:122] "RemoveContainer" containerID="80e505b32f849ddbddecd9ed7061779a2a1149897458406664a16b75c81e7533" Nov 03 04:30:24.270704 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:24.270693 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2" Nov 03 04:30:24.439629 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:24.439583 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2"] Nov 03 04:30:24.446702 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:24.446658 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894670-bhds2"] Nov 03 04:30:25.088975 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:30:25.088943 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" path="/var/lib/kubelet/pods/8a118a1a-7eb8-4032-aa7b-341b27d0fdb9/volumes" Nov 03 04:30:30.011727 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:30:30.011680 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:31:00.009930 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:31:00.009887 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:31:07.917138 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:07.917023 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:31:07.924385 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:07.922808 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:31:07.934987 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:07.934966 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:31:07.939661 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:07.939641 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:31:27.965462 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.965412 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45"] Nov 03 04:31:27.966006 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.965835 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:27.966006 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.965849 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:27.966006 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.965869 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:27.966006 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.965874 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:27.966006 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.965946 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:27.966006 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.965960 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:27.969115 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:27.969099 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.027697 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.027660 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45"] Nov 03 04:31:28.075353 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.075319 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:31:28.075353 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.075341 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:31:28.079485 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.079426 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg948\" (UniqueName: \"kubernetes.io/projected/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-kube-api-access-wg948\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.079647 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.079496 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.079647 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.079622 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.180924 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.180875 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg948\" (UniqueName: \"kubernetes.io/projected/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-kube-api-access-wg948\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.181150 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.180956 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.181150 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.181024 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.181694 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.181669 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.183614 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.183595 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.242873 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.242788 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg948\" (UniqueName: \"kubernetes.io/projected/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-kube-api-access-wg948\") pod \"maas-api-revoke-sub-2d07d5a5-w7r45\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.282201 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.282168 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:28.659060 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:28.659028 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45"] Nov 03 04:31:28.660267 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:31:28.660238 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2fb6b8_2d3f_4f5f_89dc_2d0870fa45ac.slice/crio-ebdeb6416f0a11d2617df6357083eae404f2d9f2fef49a528605a43b23f27c43 WatchSource:0}: Error finding container ebdeb6416f0a11d2617df6357083eae404f2d9f2fef49a528605a43b23f27c43: Status 404 returned error can't find the container with id ebdeb6416f0a11d2617df6357083eae404f2d9f2fef49a528605a43b23f27c43 Nov 03 04:31:29.652956 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:29.652922 5117 generic.go:356] "Generic (PLEG): container finished" podID="0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" containerID="6572639fef24f39da6f605f94e912b539a9c074c00336943106d472ff6eb8fcc" exitCode=0 Nov 03 04:31:29.653362 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:29.653005 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" event={"ID":"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac","Type":"ContainerDied","Data":"6572639fef24f39da6f605f94e912b539a9c074c00336943106d472ff6eb8fcc"} Nov 03 04:31:29.653362 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:29.653060 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" event={"ID":"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac","Type":"ContainerStarted","Data":"ebdeb6416f0a11d2617df6357083eae404f2d9f2fef49a528605a43b23f27c43"} Nov 03 04:31:30.011642 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:31:30.011602 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:31:30.803920 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:30.803887 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:30.905375 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:30.905339 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-kube-api-access-wg948\" (UniqueName: \"kubernetes.io/projected/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-kube-api-access-wg948\") pod \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " Nov 03 04:31:30.905614 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:30.905477 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-xks-ca\") pod \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " Nov 03 04:31:30.905614 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:30.905574 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-openshift-ca\") pod \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\" (UID: \"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac\") " Nov 03 04:31:30.905929 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:30.905899 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-openshift-ca" pod "0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" (UID: "0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:31:30.907712 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:30.907688 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-xks-ca" pod "0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" (UID: "0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:31:30.907846 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:30.907825 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-kube-api-access-wg948" pod "0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" (UID: "0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac"). InnerVolumeSpecName "kube-api-access-wg948". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:31:31.007354 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:31.007311 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg948\" (UniqueName: \"kubernetes.io/projected/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-kube-api-access-wg948\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:31:31.007354 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:31.007352 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-xks-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:31:31.007354 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:31.007362 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac-maas-api-openshift-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:31:31.664914 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:31.664876 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" event={"ID":"0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac","Type":"ContainerDied","Data":"ebdeb6416f0a11d2617df6357083eae404f2d9f2fef49a528605a43b23f27c43"} Nov 03 04:31:31.665111 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:31.664925 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebdeb6416f0a11d2617df6357083eae404f2d9f2fef49a528605a43b23f27c43" Nov 03 04:31:31.665111 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:31.664891 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45" Nov 03 04:31:46.801118 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.801016 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk"] Nov 03 04:31:46.801833 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.801668 5117 cpu_manager.go:414] "containerMap: removing container" podUID="0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" containerName="revoke-keys" Nov 03 04:31:46.801833 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.801690 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" containerName="revoke-keys" Nov 03 04:31:46.801833 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.801732 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:46.801833 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.801741 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:46.802085 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.801874 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" containerName="revoke-keys" Nov 03 04:31:46.802085 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.801892 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a118a1a-7eb8-4032-aa7b-341b27d0fdb9" containerName="cleanup" Nov 03 04:31:46.805546 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.805518 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.847725 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.847664 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.847725 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.847727 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46lj\" (UniqueName: \"kubernetes.io/projected/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-kube-api-access-m46lj\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.848041 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.847881 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.863715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.863672 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk"] Nov 03 04:31:46.910027 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.909986 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 04:31:46.910261 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.910193 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zddpg\"" Nov 03 04:31:46.948381 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.948341 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.948381 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.948390 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46lj\" (UniqueName: \"kubernetes.io/projected/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-kube-api-access-m46lj\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.948730 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.948481 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.949364 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.949332 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:46.951233 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:46.951205 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:47.011468 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:47.011419 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46lj\" (UniqueName: \"kubernetes.io/projected/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-kube-api-access-m46lj\") pod \"maas-api-revoke-sub-08de66a8-2bqvk\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:47.124477 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:47.124409 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:47.526791 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:47.526369 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk"] Nov 03 04:31:47.584371 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:31:47.584332 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb840eaa9_001d_4f50_aede_0a5f3dcd7c61.slice/crio-f3e2a4e7cd57ac35decb81489ca6744f530c4880c210d83ac322d23ce6f61644 WatchSource:0}: Error finding container f3e2a4e7cd57ac35decb81489ca6744f530c4880c210d83ac322d23ce6f61644: Status 404 returned error can't find the container with id f3e2a4e7cd57ac35decb81489ca6744f530c4880c210d83ac322d23ce6f61644 Nov 03 04:31:47.769868 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:47.769811 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" event={"ID":"b840eaa9-001d-4f50-aede-0a5f3dcd7c61","Type":"ContainerStarted","Data":"f3e2a4e7cd57ac35decb81489ca6744f530c4880c210d83ac322d23ce6f61644"} Nov 03 04:31:48.785917 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:48.785882 5117 generic.go:356] "Generic (PLEG): container finished" podID="b840eaa9-001d-4f50-aede-0a5f3dcd7c61" containerID="46f5c982523fcd7d2cace0b24420188794be4d1169454c802853b9087797dccb" exitCode=0 Nov 03 04:31:48.786347 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:48.785958 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" event={"ID":"b840eaa9-001d-4f50-aede-0a5f3dcd7c61","Type":"ContainerDied","Data":"46f5c982523fcd7d2cace0b24420188794be4d1169454c802853b9087797dccb"} Nov 03 04:31:49.933279 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:49.933251 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:50.001009 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.000949 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-kube-api-access-m46lj\" (UniqueName: \"kubernetes.io/projected/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-kube-api-access-m46lj\") pod \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " Nov 03 04:31:50.001242 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.001114 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-openshift-ca\") pod \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " Nov 03 04:31:50.001242 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.001152 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-xks-ca\") pod \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\" (UID: \"b840eaa9-001d-4f50-aede-0a5f3dcd7c61\") " Nov 03 04:31:50.001596 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.001567 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-openshift-ca" pod "b840eaa9-001d-4f50-aede-0a5f3dcd7c61" (UID: "b840eaa9-001d-4f50-aede-0a5f3dcd7c61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:31:50.003500 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.003463 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-xks-ca" pod "b840eaa9-001d-4f50-aede-0a5f3dcd7c61" (UID: "b840eaa9-001d-4f50-aede-0a5f3dcd7c61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:31:50.003500 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.003475 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-kube-api-access-m46lj" pod "b840eaa9-001d-4f50-aede-0a5f3dcd7c61" (UID: "b840eaa9-001d-4f50-aede-0a5f3dcd7c61"). InnerVolumeSpecName "kube-api-access-m46lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:31:50.102268 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.102223 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-openshift-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:31:50.102268 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.102261 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-maas-api-xks-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:31:50.102268 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.102270 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46lj\" (UniqueName: \"kubernetes.io/projected/b840eaa9-001d-4f50-aede-0a5f3dcd7c61-kube-api-access-m46lj\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:31:50.797134 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.797103 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" Nov 03 04:31:50.797325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.797102 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk" event={"ID":"b840eaa9-001d-4f50-aede-0a5f3dcd7c61","Type":"ContainerDied","Data":"f3e2a4e7cd57ac35decb81489ca6744f530c4880c210d83ac322d23ce6f61644"} Nov 03 04:31:50.797325 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:31:50.797231 5117 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e2a4e7cd57ac35decb81489ca6744f530c4880c210d83ac322d23ce6f61644" Nov 03 04:32:00.011203 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:32:00.011162 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:32:05.510473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:05.507568 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:32:05.510473 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:05.507645 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:32:30.010684 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:32:30.010640 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:32:34.336344 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.336302 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj"] Nov 03 04:32:34.336970 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.336709 5117 cpu_manager.go:414] "containerMap: removing container" podUID="b840eaa9-001d-4f50-aede-0a5f3dcd7c61" containerName="revoke-keys" Nov 03 04:32:34.336970 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.336720 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b840eaa9-001d-4f50-aede-0a5f3dcd7c61" containerName="revoke-keys" Nov 03 04:32:34.336970 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.336802 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="b840eaa9-001d-4f50-aede-0a5f3dcd7c61" containerName="revoke-keys" Nov 03 04:32:34.340168 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.340148 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.417711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.417675 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj"] Nov 03 04:32:34.436621 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.436588 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-model-cache\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.436776 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.436728 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62jt\" (UniqueName: \"kubernetes.io/projected/15ae5db0-49f9-4363-a205-a003139aca4e-kube-api-access-v62jt\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.436817 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.436781 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-dshm\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.436853 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.436830 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae5db0-49f9-4363-a205-a003139aca4e-tls-certs\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.436891 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.436855 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-home\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.436928 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.436890 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-tmp-dir\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.448806 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.448776 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-deca42\"/\"simulator-single-main-deca42-kserve-self-signed-certs\"" Nov 03 04:32:34.448806 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.448779 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-deca42\"/\"openshift-service-ca.crt\"" Nov 03 04:32:34.449010 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.448840 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-deca42\"/\"default-dockercfg-tr6zn\"" Nov 03 04:32:34.449010 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.448867 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-deca42\"/\"kube-root-ca.crt\"" Nov 03 04:32:34.537718 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.537686 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-tmp-dir\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.537918 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.537739 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-model-cache\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.537918 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.537790 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62jt\" (UniqueName: \"kubernetes.io/projected/15ae5db0-49f9-4363-a205-a003139aca4e-kube-api-access-v62jt\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.537918 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.537810 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-dshm\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.538099 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.537938 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae5db0-49f9-4363-a205-a003139aca4e-tls-certs\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.538099 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.537988 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-home\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.538215 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.538181 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-tmp-dir\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.538254 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.538229 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-model-cache\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.538395 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.538368 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-home\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.540466 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.540423 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-dshm\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.540637 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.540618 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae5db0-49f9-4363-a205-a003139aca4e-tls-certs\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.600555 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.600476 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62jt\" (UniqueName: \"kubernetes.io/projected/15ae5db0-49f9-4363-a205-a003139aca4e-kube-api-access-v62jt\") pod \"simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:34.651580 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:34.651549 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:35.042305 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:35.042273 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj"] Nov 03 04:32:35.043556 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:32:35.043528 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15ae5db0_49f9_4363_a205_a003139aca4e.slice/crio-5148d351ead84ce3fd434a334f34a944e7babba95e14a70f8e77dcc9e6c070d4 WatchSource:0}: Error finding container 5148d351ead84ce3fd434a334f34a944e7babba95e14a70f8e77dcc9e6c070d4: Status 404 returned error can't find the container with id 5148d351ead84ce3fd434a334f34a944e7babba95e14a70f8e77dcc9e6c070d4 Nov 03 04:32:36.062150 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:36.062109 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" event={"ID":"15ae5db0-49f9-4363-a205-a003139aca4e","Type":"ContainerStarted","Data":"2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c"} Nov 03 04:32:36.062715 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:36.062164 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" event={"ID":"15ae5db0-49f9-4363-a205-a003139aca4e","Type":"ContainerStarted","Data":"5148d351ead84ce3fd434a334f34a944e7babba95e14a70f8e77dcc9e6c070d4"} Nov 03 04:32:36.181773 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:36.181717 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" podStartSLOduration=2.181705172 podStartE2EDuration="2.181705172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:32:36.1801831 +0000 UTC m=+5491.210781637" watchObservedRunningTime="2026-11-03 04:32:36.181705172 +0000 UTC m=+5491.212303709" Nov 03 04:32:36.652656 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:36.652615 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:44.652140 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:44.652092 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:44.664898 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:44.664874 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:32:45.120357 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:32:45.120328 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:33:00.012352 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:33:00.012314 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:33:30.011026 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:33:30.010931 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:33:35.126964 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:35.126932 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj"] Nov 03 04:33:50.147416 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.147352 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" podUID="15ae5db0-49f9-4363-a205-a003139aca4e" containerName="main" containerID="cri-o://2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c" gracePeriod=45 Nov 03 04:33:50.394167 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.394141 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:33:50.448222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.448145 5117 generic.go:356] "Generic (PLEG): container finished" podID="15ae5db0-49f9-4363-a205-a003139aca4e" containerID="2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c" exitCode=0 Nov 03 04:33:50.448222 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.448192 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" Nov 03 04:33:50.448401 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.448235 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" event={"ID":"15ae5db0-49f9-4363-a205-a003139aca4e","Type":"ContainerDied","Data":"2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c"} Nov 03 04:33:50.448401 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.448289 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj" event={"ID":"15ae5db0-49f9-4363-a205-a003139aca4e","Type":"ContainerDied","Data":"5148d351ead84ce3fd434a334f34a944e7babba95e14a70f8e77dcc9e6c070d4"} Nov 03 04:33:50.448401 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.448312 5117 scope.go:122] "RemoveContainer" containerID="2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c" Nov 03 04:33:50.460138 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.460115 5117 scope.go:122] "RemoveContainer" containerID="2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c" Nov 03 04:33:50.460422 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:33:50.460399 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c\": container with ID starting with 2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c not found: ID does not exist" containerID="2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c" Nov 03 04:33:50.460549 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.460435 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c"} err="failed to get container status \"2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c\": rpc error: code = NotFound desc = could not find container \"2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c\": container with ID starting with 2ca812dd55524b9edbc49eb1ba0b3b8438ce421b5caef6019aef4cd27aa6797c not found: ID does not exist" Nov 03 04:33:50.537070 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537040 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-home\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-home\") pod \"15ae5db0-49f9-4363-a205-a003139aca4e\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " Nov 03 04:33:50.537274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537102 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-dshm\") pod \"15ae5db0-49f9-4363-a205-a003139aca4e\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " Nov 03 04:33:50.537274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537130 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-model-cache\") pod \"15ae5db0-49f9-4363-a205-a003139aca4e\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " Nov 03 04:33:50.537274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537156 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-tmp-dir\") pod \"15ae5db0-49f9-4363-a205-a003139aca4e\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " Nov 03 04:33:50.537274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537271 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ae5db0-49f9-4363-a205-a003139aca4e-kube-api-access-v62jt\" (UniqueName: \"kubernetes.io/projected/15ae5db0-49f9-4363-a205-a003139aca4e-kube-api-access-v62jt\") pod \"15ae5db0-49f9-4363-a205-a003139aca4e\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " Nov 03 04:33:50.537560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537362 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-home" pod "15ae5db0-49f9-4363-a205-a003139aca4e" (UID: "15ae5db0-49f9-4363-a205-a003139aca4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:33:50.537560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537460 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-model-cache" pod "15ae5db0-49f9-4363-a205-a003139aca4e" (UID: "15ae5db0-49f9-4363-a205-a003139aca4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:33:50.537560 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537309 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ae5db0-49f9-4363-a205-a003139aca4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae5db0-49f9-4363-a205-a003139aca4e-tls-certs\") pod \"15ae5db0-49f9-4363-a205-a003139aca4e\" (UID: \"15ae5db0-49f9-4363-a205-a003139aca4e\") " Nov 03 04:33:50.537714 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537581 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-tmp-dir" pod "15ae5db0-49f9-4363-a205-a003139aca4e" (UID: "15ae5db0-49f9-4363-a205-a003139aca4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:33:50.537914 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.537890 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:50.538240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.538218 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:50.538345 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.538245 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:50.539555 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.539534 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-dshm" pod "15ae5db0-49f9-4363-a205-a003139aca4e" (UID: "15ae5db0-49f9-4363-a205-a003139aca4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:33:50.539651 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.539628 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ae5db0-49f9-4363-a205-a003139aca4e-tls-certs" pod "15ae5db0-49f9-4363-a205-a003139aca4e" (UID: "15ae5db0-49f9-4363-a205-a003139aca4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:33:50.539748 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.539723 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ae5db0-49f9-4363-a205-a003139aca4e-kube-api-access-v62jt" pod "15ae5db0-49f9-4363-a205-a003139aca4e" (UID: "15ae5db0-49f9-4363-a205-a003139aca4e"). InnerVolumeSpecName "kube-api-access-v62jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:33:50.639240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.639205 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ae5db0-49f9-4363-a205-a003139aca4e-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:50.639240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.639237 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v62jt\" (UniqueName: \"kubernetes.io/projected/15ae5db0-49f9-4363-a205-a003139aca4e-kube-api-access-v62jt\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:50.639240 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.639248 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ae5db0-49f9-4363-a205-a003139aca4e-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:33:50.871059 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.871018 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj"] Nov 03 04:33:50.875565 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:50.875535 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-deca42/simulator-single-main-deca42-kserve-7bd9f4bff-gkkwj"] Nov 03 04:33:51.089340 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:33:51.089302 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ae5db0-49f9-4363-a205-a003139aca4e" path="/var/lib/kubelet/pods/15ae5db0-49f9-4363-a205-a003139aca4e/volumes" Nov 03 04:34:00.010391 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:34:00.010349 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:34:30.010847 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:34:30.010803 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:35:00.010872 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:35:00.010829 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:35:29.671633 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:35:29.671585 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:35:29.672310 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:35:29.672263 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:35:30.080299 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:35:30.080251 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:36:00.010413 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:36:00.010375 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:36:07.959180 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:07.959047 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:36:07.965767 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:07.965735 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:36:07.983309 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:07.983275 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:36:07.990165 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:07.990136 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:36:30.011841 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:36:30.011801 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:36:31.051608 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:31.051568 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45"] Nov 03 04:36:31.101977 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:31.101945 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d07d5a5-w7r45"] Nov 03 04:36:33.089286 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:33.089245 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac" path="/var/lib/kubelet/pods/0a2fb6b8-2d3f-4f5f-89dc-2d0870fa45ac/volumes" Nov 03 04:36:51.049706 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:51.049670 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk"] Nov 03 04:36:51.100126 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:51.100083 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08de66a8-2bqvk"] Nov 03 04:36:53.088435 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:36:53.088398 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b840eaa9-001d-4f50-aede-0a5f3dcd7c61" path="/var/lib/kubelet/pods/b840eaa9-001d-4f50-aede-0a5f3dcd7c61/volumes" Nov 03 04:37:00.009885 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:37:00.009847 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:37:30.011406 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:37:30.011355 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:38:00.010121 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:38:00.010073 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:38:07.760826 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:38:07.760794 5117 scope.go:122] "RemoveContainer" containerID="6572639fef24f39da6f605f94e912b539a9c074c00336943106d472ff6eb8fcc" Nov 03 04:38:07.769990 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:38:07.769968 5117 scope.go:122] "RemoveContainer" containerID="46f5c982523fcd7d2cace0b24420188794be4d1169454c802853b9087797dccb" Nov 03 04:38:30.010557 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:38:30.010516 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:39:00.010777 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:39:00.010738 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:39:30.011488 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:39:30.011379 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:39:44.673600 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:39:44.673554 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:39:44.674120 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:39:44.673632 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:40:00.010837 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:40:00.010795 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:40:30.012178 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:40:30.012140 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:41:00.010877 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:41:00.010838 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:41:08.008153 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:41:08.008041 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:41:08.013144 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:41:08.013118 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:41:08.023213 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:41:08.023193 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:41:08.027740 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:41:08.027725 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:41:30.011101 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:41:30.011058 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:42:00.011154 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:42:00.011108 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:42:30.010652 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:42:30.010567 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:42:50.495680 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:42:50.495633 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:42:50.496192 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:42:50.495711 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:43:00.011295 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:43:00.011246 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:43:30.011844 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:43:30.011806 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:44:00.011694 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:44:00.011647 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:44:30.011478 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:44:30.011418 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:45:00.014598 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:45:00.014553 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:45:24.054368 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.054274 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d"] Nov 03 04:45:24.054906 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.054771 5117 cpu_manager.go:414] "containerMap: removing container" podUID="15ae5db0-49f9-4363-a205-a003139aca4e" containerName="main" Nov 03 04:45:24.054906 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.054787 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ae5db0-49f9-4363-a205-a003139aca4e" containerName="main" Nov 03 04:45:24.054906 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.054886 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ae5db0-49f9-4363-a205-a003139aca4e" containerName="main" Nov 03 04:45:24.057944 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.057925 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.139017 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.138971 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea7503d-cf17-475a-863f-98d86acb76f1-tls-certs\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.139209 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.139039 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-model-cache\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.139209 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.139165 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-dshm\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.139320 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.139253 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-tmp-dir\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.139393 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.139365 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8xj\" (UniqueName: \"kubernetes.io/projected/eea7503d-cf17-475a-863f-98d86acb76f1-kube-api-access-9r8xj\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.139502 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.139485 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-home\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.139965 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.139945 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d"] Nov 03 04:45:24.165026 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.164991 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bcb427\"/\"kube-root-ca.crt\"" Nov 03 04:45:24.165229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.165031 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bcb427\"/\"openshift-service-ca.crt\"" Nov 03 04:45:24.165229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.165036 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bcb427\"/\"default-dockercfg-ntl25\"" Nov 03 04:45:24.165229 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.165097 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bcb427\"/\"simulator-single-main-bcb427-kserve-self-signed-certs\"" Nov 03 04:45:24.240860 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.240829 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-home\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.240884 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea7503d-cf17-475a-863f-98d86acb76f1-tls-certs\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.240915 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-model-cache\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.240940 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-dshm\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.240975 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-tmp-dir\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.240996 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8xj\" (UniqueName: \"kubernetes.io/projected/eea7503d-cf17-475a-863f-98d86acb76f1-kube-api-access-9r8xj\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241336 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.241317 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-home\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241400 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.241370 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-model-cache\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.241484 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.241424 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-tmp-dir\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.243492 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.243466 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-dshm\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.243689 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.243674 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea7503d-cf17-475a-863f-98d86acb76f1-tls-certs\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.303728 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.303689 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8xj\" (UniqueName: \"kubernetes.io/projected/eea7503d-cf17-475a-863f-98d86acb76f1-kube-api-access-9r8xj\") pod \"simulator-single-main-bcb427-kserve-76978c6656-n557d\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.370138 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.370100 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:24.750592 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:24.750561 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d"] Nov 03 04:45:24.752872 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:45:24.752836 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeea7503d_cf17_475a_863f_98d86acb76f1.slice/crio-bdc6258f84a6c870f72857697cf80f7ddaef0a3cbc00362b280decdf5dccc303 WatchSource:0}: Error finding container bdc6258f84a6c870f72857697cf80f7ddaef0a3cbc00362b280decdf5dccc303: Status 404 returned error can't find the container with id bdc6258f84a6c870f72857697cf80f7ddaef0a3cbc00362b280decdf5dccc303 Nov 03 04:45:25.469868 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:25.469814 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" event={"ID":"eea7503d-cf17-475a-863f-98d86acb76f1","Type":"ContainerStarted","Data":"aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5"} Nov 03 04:45:25.469868 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:25.469873 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" event={"ID":"eea7503d-cf17-475a-863f-98d86acb76f1","Type":"ContainerStarted","Data":"bdc6258f84a6c870f72857697cf80f7ddaef0a3cbc00362b280decdf5dccc303"} Nov 03 04:45:25.594089 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:25.594026 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" podStartSLOduration=1.594012443 podStartE2EDuration="1.594012443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:45:25.591917649 +0000 UTC m=+6260.622516186" watchObservedRunningTime="2026-11-03 04:45:25.594012443 +0000 UTC m=+6260.624610980" Nov 03 04:45:26.370717 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:26.370677 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:30.012306 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:45:30.012268 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:45:34.371183 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:34.371139 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:34.383963 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:34.383933 5117 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:45:34.526357 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:45:34.526323 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:46:00.009963 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:46:00.009919 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:46:08.042797 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:08.042681 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:46:08.048044 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:08.048021 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:46:08.059574 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:08.059553 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:46:08.064677 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:08.064659 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:46:24.378298 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:24.378261 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d"] Nov 03 04:46:30.010374 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:46:30.010336 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:46:39.398815 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.398702 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" podUID="eea7503d-cf17-475a-863f-98d86acb76f1" containerName="main" containerID="cri-o://aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5" gracePeriod=45 Nov 03 04:46:39.516537 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.516490 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" podUID="eea7503d-cf17-475a-863f-98d86acb76f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.149:8000/ready\": dial tcp 10.132.0.149:8000: connect: connection refused" Nov 03 04:46:39.645437 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.645407 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:46:39.773478 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773384 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-home\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-home\") pod \"eea7503d-cf17-475a-863f-98d86acb76f1\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " Nov 03 04:46:39.773478 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773438 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-dshm\") pod \"eea7503d-cf17-475a-863f-98d86acb76f1\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " Nov 03 04:46:39.773711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773516 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea7503d-cf17-475a-863f-98d86acb76f1-kube-api-access-9r8xj\" (UniqueName: \"kubernetes.io/projected/eea7503d-cf17-475a-863f-98d86acb76f1-kube-api-access-9r8xj\") pod \"eea7503d-cf17-475a-863f-98d86acb76f1\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " Nov 03 04:46:39.773711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773590 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-model-cache\") pod \"eea7503d-cf17-475a-863f-98d86acb76f1\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " Nov 03 04:46:39.773711 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773685 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea7503d-cf17-475a-863f-98d86acb76f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/eea7503d-cf17-475a-863f-98d86acb76f1-tls-certs\") pod \"eea7503d-cf17-475a-863f-98d86acb76f1\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " Nov 03 04:46:39.773886 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773712 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-home" pod "eea7503d-cf17-475a-863f-98d86acb76f1" (UID: "eea7503d-cf17-475a-863f-98d86acb76f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:46:39.773886 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773723 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-tmp-dir\") pod \"eea7503d-cf17-475a-863f-98d86acb76f1\" (UID: \"eea7503d-cf17-475a-863f-98d86acb76f1\") " Nov 03 04:46:39.773977 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773899 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-model-cache" pod "eea7503d-cf17-475a-863f-98d86acb76f1" (UID: "eea7503d-cf17-475a-863f-98d86acb76f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:46:39.773977 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.773957 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-tmp-dir" pod "eea7503d-cf17-475a-863f-98d86acb76f1" (UID: "eea7503d-cf17-475a-863f-98d86acb76f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:46:39.774204 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.774186 5117 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-model-cache\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:39.774274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.774211 5117 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-tmp-dir\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:39.774274 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.774224 5117 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-home\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:39.777050 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.777013 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea7503d-cf17-475a-863f-98d86acb76f1-kube-api-access-9r8xj" pod "eea7503d-cf17-475a-863f-98d86acb76f1" (UID: "eea7503d-cf17-475a-863f-98d86acb76f1"). InnerVolumeSpecName "kube-api-access-9r8xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:46:39.777197 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.777173 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea7503d-cf17-475a-863f-98d86acb76f1-tls-certs" pod "eea7503d-cf17-475a-863f-98d86acb76f1" (UID: "eea7503d-cf17-475a-863f-98d86acb76f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:46:39.780930 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.780904 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-dshm" pod "eea7503d-cf17-475a-863f-98d86acb76f1" (UID: "eea7503d-cf17-475a-863f-98d86acb76f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 04:46:39.846764 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.846731 5117 generic.go:356] "Generic (PLEG): container finished" podID="eea7503d-cf17-475a-863f-98d86acb76f1" containerID="aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5" exitCode=0 Nov 03 04:46:39.846939 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.846783 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" Nov 03 04:46:39.846939 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.846784 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" event={"ID":"eea7503d-cf17-475a-863f-98d86acb76f1","Type":"ContainerDied","Data":"aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5"} Nov 03 04:46:39.846939 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.846888 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d" event={"ID":"eea7503d-cf17-475a-863f-98d86acb76f1","Type":"ContainerDied","Data":"bdc6258f84a6c870f72857697cf80f7ddaef0a3cbc00362b280decdf5dccc303"} Nov 03 04:46:39.846939 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.846904 5117 scope.go:122] "RemoveContainer" containerID="aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5" Nov 03 04:46:39.857115 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.857090 5117 scope.go:122] "RemoveContainer" containerID="aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5" Nov 03 04:46:39.857426 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:46:39.857406 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5\": container with ID starting with aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5 not found: ID does not exist" containerID="aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5" Nov 03 04:46:39.857550 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.857522 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5"} err="failed to get container status \"aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5\": rpc error: code = NotFound desc = could not find container \"aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5\": container with ID starting with aa2b5d542fcfb7717eefee1951902371133c02790d6ed7359db377935e3e7ab5 not found: ID does not exist" Nov 03 04:46:39.875012 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.874990 5117 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea7503d-cf17-475a-863f-98d86acb76f1-tls-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:39.875110 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.875013 5117 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea7503d-cf17-475a-863f-98d86acb76f1-dshm\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:39.875110 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:39.875031 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8xj\" (UniqueName: \"kubernetes.io/projected/eea7503d-cf17-475a-863f-98d86acb76f1-kube-api-access-9r8xj\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:46:40.016333 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:40.016292 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d"] Nov 03 04:46:40.023110 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:40.023081 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-bcb427/simulator-single-main-bcb427-kserve-76978c6656-n557d"] Nov 03 04:46:41.089658 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:46:41.089623 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea7503d-cf17-475a-863f-98d86acb76f1" path="/var/lib/kubelet/pods/eea7503d-cf17-475a-863f-98d86acb76f1/volumes" Nov 03 04:47:00.012286 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:47:00.012244 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:47:30.010321 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:47:30.010273 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:48:00.010523 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:48:00.010482 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:48:30.010163 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:48:30.010075 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:49:00.011249 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:49:00.011204 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:49:28.768931 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.768890 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq"] Nov 03 04:49:28.769585 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.769565 5117 cpu_manager.go:414] "containerMap: removing container" podUID="eea7503d-cf17-475a-863f-98d86acb76f1" containerName="main" Nov 03 04:49:28.769678 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.769585 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea7503d-cf17-475a-863f-98d86acb76f1" containerName="main" Nov 03 04:49:28.769738 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.769726 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea7503d-cf17-475a-863f-98d86acb76f1" containerName="main" Nov 03 04:49:28.772780 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.772759 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.818040 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.818008 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ptb\" (UniqueName: \"kubernetes.io/projected/8671112d-396a-401d-a6be-71f0c61cccaf-kube-api-access-d4ptb\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.818252 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.818079 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-metrics-certs\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.818252 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.818127 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8671112d-396a-401d-a6be-71f0c61cccaf-openshift-service-ca\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.818380 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.818280 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-maas-api-tls\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.880693 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.880524 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 04:49:28.880693 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.880584 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcxk\"" Nov 03 04:49:28.881173 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.881149 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 04:49:28.885782 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.885753 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq"] Nov 03 04:49:28.919182 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.919146 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ptb\" (UniqueName: \"kubernetes.io/projected/8671112d-396a-401d-a6be-71f0c61cccaf-kube-api-access-d4ptb\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.919376 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.919204 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-metrics-certs\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.919376 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.919221 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8671112d-396a-401d-a6be-71f0c61cccaf-openshift-service-ca\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.919376 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.919269 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-maas-api-tls\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.920117 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.920038 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8671112d-396a-401d-a6be-71f0c61cccaf-openshift-service-ca\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.921854 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.921831 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-metrics-certs\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.922005 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.921986 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-maas-api-tls\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:28.982153 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:28.982119 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ptb\" (UniqueName: \"kubernetes.io/projected/8671112d-396a-401d-a6be-71f0c61cccaf-kube-api-access-d4ptb\") pod \"maas-api-55cd9fdc56-tf5dq\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:29.085279 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:29.085250 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:29.463732 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:29.463700 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq"] Nov 03 04:49:29.465912 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:49:29.465880 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8671112d_396a_401d_a6be_71f0c61cccaf.slice/crio-a9e28cd6f376b7b74752333f9fed4e5b6ff32b04d42a8caef63dce75dcbdf29e WatchSource:0}: Error finding container a9e28cd6f376b7b74752333f9fed4e5b6ff32b04d42a8caef63dce75dcbdf29e: Status 404 returned error can't find the container with id a9e28cd6f376b7b74752333f9fed4e5b6ff32b04d42a8caef63dce75dcbdf29e Nov 03 04:49:29.468133 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:29.468106 5117 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 04:49:29.770655 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:29.770571 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" event={"ID":"8671112d-396a-401d-a6be-71f0c61cccaf","Type":"ContainerStarted","Data":"a9e28cd6f376b7b74752333f9fed4e5b6ff32b04d42a8caef63dce75dcbdf29e"} Nov 03 04:49:30.009832 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:49:30.009797 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:49:30.777092 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:30.776998 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" event={"ID":"8671112d-396a-401d-a6be-71f0c61cccaf","Type":"ContainerStarted","Data":"1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4"} Nov 03 04:49:30.777092 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:30.777074 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:30.898437 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:30.898381 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" podStartSLOduration=1.984287969 podStartE2EDuration="2.898364963s" totalImagesPullingTime="914.076994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:49:30.896298807 +0000 UTC m=+6505.926897345" watchObservedRunningTime="2026-11-03 04:49:30.898364963 +0000 UTC m=+6505.928963545" Nov 03 04:49:36.787323 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:36.787293 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:37.125623 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.125589 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2"] Nov 03 04:49:37.129430 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.129410 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.187541 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.187497 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2"] Nov 03 04:49:37.196427 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.196399 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx74d\" (UniqueName: \"kubernetes.io/projected/6f02b17f-dba2-4612-9eca-b7911c5ec968-kube-api-access-kx74d\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.196604 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.196462 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f02b17f-dba2-4612-9eca-b7911c5ec968-metrics-certs\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.196679 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.196619 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f02b17f-dba2-4612-9eca-b7911c5ec968-openshift-service-ca\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.196679 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.196662 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f02b17f-dba2-4612-9eca-b7911c5ec968-maas-api-tls\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.297519 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.297399 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f02b17f-dba2-4612-9eca-b7911c5ec968-openshift-service-ca\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.297519 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.297470 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f02b17f-dba2-4612-9eca-b7911c5ec968-maas-api-tls\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.297777 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.297567 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx74d\" (UniqueName: \"kubernetes.io/projected/6f02b17f-dba2-4612-9eca-b7911c5ec968-kube-api-access-kx74d\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.297777 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.297629 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f02b17f-dba2-4612-9eca-b7911c5ec968-metrics-certs\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.298227 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.298200 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f02b17f-dba2-4612-9eca-b7911c5ec968-openshift-service-ca\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.300262 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.300235 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f02b17f-dba2-4612-9eca-b7911c5ec968-metrics-certs\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.300356 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.300315 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f02b17f-dba2-4612-9eca-b7911c5ec968-maas-api-tls\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.359717 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.359687 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx74d\" (UniqueName: \"kubernetes.io/projected/6f02b17f-dba2-4612-9eca-b7911c5ec968-kube-api-access-kx74d\") pod \"maas-api-b77bddc5b-kdcw2\" (UID: \"6f02b17f-dba2-4612-9eca-b7911c5ec968\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.442500 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.442420 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:37.820388 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:37.820359 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2"] Nov 03 04:49:37.822541 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:49:37.822513 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f02b17f_dba2_4612_9eca_b7911c5ec968.slice/crio-da7f3e39c1eeaf88a3cc61c7f5523a07cb17400aae1a72b4a56ea5a86ac3ca80 WatchSource:0}: Error finding container da7f3e39c1eeaf88a3cc61c7f5523a07cb17400aae1a72b4a56ea5a86ac3ca80: Status 404 returned error can't find the container with id da7f3e39c1eeaf88a3cc61c7f5523a07cb17400aae1a72b4a56ea5a86ac3ca80 Nov 03 04:49:38.819255 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:38.819214 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" event={"ID":"6f02b17f-dba2-4612-9eca-b7911c5ec968","Type":"ContainerStarted","Data":"d8145717468e91d56362e0c58531f06bd3ee90190c2afedb9bf7f8304ea483e4"} Nov 03 04:49:38.819765 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:38.819283 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" event={"ID":"6f02b17f-dba2-4612-9eca-b7911c5ec968","Type":"ContainerStarted","Data":"da7f3e39c1eeaf88a3cc61c7f5523a07cb17400aae1a72b4a56ea5a86ac3ca80"} Nov 03 04:49:38.819765 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:38.819484 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:38.941105 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:38.940999 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" podStartSLOduration=1.143677695 podStartE2EDuration="1.940984967s" totalImagesPullingTime="797.307272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:49:38.938999591 +0000 UTC m=+6513.969598130" watchObservedRunningTime="2026-11-03 04:49:38.940984967 +0000 UTC m=+6513.971583505" Nov 03 04:49:44.830262 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:44.830233 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-kdcw2" Nov 03 04:49:45.002317 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.002283 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq"] Nov 03 04:49:45.002551 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.002512 5117 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" podUID="8671112d-396a-401d-a6be-71f0c61cccaf" containerName="maas-api" containerID="cri-o://1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4" gracePeriod=30 Nov 03 04:49:45.262865 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.262833 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:45.393938 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.393857 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8671112d-396a-401d-a6be-71f0c61cccaf-kube-api-access-d4ptb\" (UniqueName: \"kubernetes.io/projected/8671112d-396a-401d-a6be-71f0c61cccaf-kube-api-access-d4ptb\") pod \"8671112d-396a-401d-a6be-71f0c61cccaf\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " Nov 03 04:49:45.393938 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.393933 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8671112d-396a-401d-a6be-71f0c61cccaf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8671112d-396a-401d-a6be-71f0c61cccaf-openshift-service-ca\") pod \"8671112d-396a-401d-a6be-71f0c61cccaf\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " Nov 03 04:49:45.394157 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.394040 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-maas-api-tls\") pod \"8671112d-396a-401d-a6be-71f0c61cccaf\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " Nov 03 04:49:45.394157 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.394138 5117 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-metrics-certs\") pod \"8671112d-396a-401d-a6be-71f0c61cccaf\" (UID: \"8671112d-396a-401d-a6be-71f0c61cccaf\") " Nov 03 04:49:45.394291 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.394267 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8671112d-396a-401d-a6be-71f0c61cccaf-openshift-service-ca" pod "8671112d-396a-401d-a6be-71f0c61cccaf" (UID: "8671112d-396a-401d-a6be-71f0c61cccaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 04:49:45.394520 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.394499 5117 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8671112d-396a-401d-a6be-71f0c61cccaf-openshift-service-ca\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:45.396184 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.396163 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8671112d-396a-401d-a6be-71f0c61cccaf-kube-api-access-d4ptb" pod "8671112d-396a-401d-a6be-71f0c61cccaf" (UID: "8671112d-396a-401d-a6be-71f0c61cccaf"). InnerVolumeSpecName "kube-api-access-d4ptb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 04:49:45.396773 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.396748 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-maas-api-tls" pod "8671112d-396a-401d-a6be-71f0c61cccaf" (UID: "8671112d-396a-401d-a6be-71f0c61cccaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:49:45.396773 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.396762 5117 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-metrics-certs" pod "8671112d-396a-401d-a6be-71f0c61cccaf" (UID: "8671112d-396a-401d-a6be-71f0c61cccaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 04:49:45.495221 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.495185 5117 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4ptb\" (UniqueName: \"kubernetes.io/projected/8671112d-396a-401d-a6be-71f0c61cccaf-kube-api-access-d4ptb\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:45.495221 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.495216 5117 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-maas-api-tls\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:45.495221 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.495225 5117 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8671112d-396a-401d-a6be-71f0c61cccaf-metrics-certs\") on node \"ip-10-0-138-120.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 04:49:45.872218 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.872182 5117 generic.go:356] "Generic (PLEG): container finished" podID="8671112d-396a-401d-a6be-71f0c61cccaf" containerID="1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4" exitCode=0 Nov 03 04:49:45.872749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.872235 5117 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" Nov 03 04:49:45.872749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.872266 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" event={"ID":"8671112d-396a-401d-a6be-71f0c61cccaf","Type":"ContainerDied","Data":"1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4"} Nov 03 04:49:45.872749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.872317 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq" event={"ID":"8671112d-396a-401d-a6be-71f0c61cccaf","Type":"ContainerDied","Data":"a9e28cd6f376b7b74752333f9fed4e5b6ff32b04d42a8caef63dce75dcbdf29e"} Nov 03 04:49:45.872749 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.872339 5117 scope.go:122] "RemoveContainer" containerID="1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4" Nov 03 04:49:45.881951 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.881936 5117 scope.go:122] "RemoveContainer" containerID="1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4" Nov 03 04:49:45.882217 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:49:45.882200 5117 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4\": container with ID starting with 1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4 not found: ID does not exist" containerID="1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4" Nov 03 04:49:45.882259 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:45.882226 5117 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4"} err="failed to get container status \"1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4\": rpc error: code = NotFound desc = could not find container \"1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4\": container with ID starting with 1f96507c00cf55da9f0557dcf062bc8ab518be6e6f3ecf34a606061e160184d4 not found: ID does not exist" Nov 03 04:49:46.040597 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:46.040561 5117 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq"] Nov 03 04:49:46.091283 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:46.091252 5117 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-tf5dq"] Nov 03 04:49:47.088823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:49:47.088779 5117 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8671112d-396a-401d-a6be-71f0c61cccaf" path="/var/lib/kubelet/pods/8671112d-396a-401d-a6be-71f0c61cccaf/volumes" Nov 03 04:50:00.010057 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:50:00.010018 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:50:30.010467 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:50:30.010409 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:51:00.011002 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:51:00.010956 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:51:08.094438 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:51:08.094410 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:51:08.099286 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:51:08.099261 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:51:08.107469 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:51:08.099509 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:51:08.107761 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:51:08.105571 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:51:30.010494 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:51:30.010434 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:52:00.011115 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:52:00.011069 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:52:05.511471 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:52:05.506818 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:52:05.511471 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:52:05.506933 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:52:30.010998 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:52:30.010959 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:52:50.499758 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:52:50.499627 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:52:50.500695 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:52:50.500645 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:53:00.010332 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:53:00.010295 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:53:14.678203 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:53:14.678152 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:53:14.678808 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:53:14.678228 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:53:30.009385 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:53:30.009342 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:53:44.677063 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:53:44.676585 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9znf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:53:44.677063 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:53:44.676650 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9znf5" podUID="b15dddef-9e9b-4662-a65f-614203a4e981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:54:00.009968 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:54:00.009930 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:54:28.479030 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:54:28.478647 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lbdzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:54:28.479030 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:54:28.478740 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lbdzp" podUID="21acc925-025e-4dd2-bd7e-3314f7337a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:54:30.011365 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:54:30.011326 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:55:00.010493 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:55:00.010437 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:55:20.510472 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:55:20.503544 5117 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rnx7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 04:55:20.510472 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:55:20.503615 5117 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rnx7l" podUID="3bc1ec57-9eb0-4153-a266-cc11de2854a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 04:55:30.076569 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:55:30.076526 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:56:00.010214 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:56:00.010174 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:56:08.145690 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:08.145577 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:56:08.151172 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:08.151151 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:56:08.153318 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:08.153297 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-6t2pr_2469c20c-30a7-41bb-b726-d62d9324f693/cluster-samples-operator/0.log" Nov 03 04:56:08.158357 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:08.158339 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-f869c_297e0916-b3bb-4002-810c-390a724cd782/console-operator/3.log" Nov 03 04:56:13.277156 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:13.277122 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jnblx_0613f917-8742-408c-80ed-640e913c05f6/manager/0.log" Nov 03 04:56:14.651682 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:14.651646 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mn4ss_636bbe26-5388-49da-93d3-7d53a66b7d94/manager/0.log" Nov 03 04:56:15.024510 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:15.024422 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pzvwf_9699142d-1bf4-46a4-9705-f5bd4c028ae6/proxy/0.log" Nov 03 04:56:16.449311 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:16.449280 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b77bddc5b-kdcw2_6f02b17f-dba2-4612-9eca-b7911c5ec968/maas-api/0.log" Nov 03 04:56:20.509493 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:20.509440 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-t94jx_568412cb-2ac5-4a39-a98f-92248dbfc164/postgres/0.log" Nov 03 04:56:23.001214 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:23.001183 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9szxj_70c7e0ef-af42-493c-baf8-1322dcbd8074/manager/0.log" Nov 03 04:56:26.188414 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:26.188380 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5cd8cf48-2vqdw_dfba0d42-1654-48c7-b958-e496bb4b94a3/kube-auth-proxy/0.log" Nov 03 04:56:27.179765 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:27.179729 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-9znf5_b15dddef-9e9b-4662-a65f-614203a4e981/istio-proxy/0.log" Nov 03 04:56:28.076180 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:28.076146 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-lbdzp_21acc925-025e-4dd2-bd7e-3314f7337a45/istio-proxy/0.log" Nov 03 04:56:28.926852 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:28.926820 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-rnx7l_3bc1ec57-9eb0-4153-a266-cc11de2854a2/istio-proxy/0.log" Nov 03 04:56:30.010174 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:56:30.010134 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:56:30.107531 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:30.107482 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c5c7c7bf7-x5tt5_c02e7385-46cc-46a6-822f-f47fe2819a3f/router/0.log" Nov 03 04:56:53.413728 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:53.413678 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j76qm_68969cec-a620-4bd5-bfd1-cd203d0822b8/global-pull-secret-syncer/0.log" Nov 03 04:56:54.611357 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:54.611326 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5r574_50ae62be-5346-4aff-b350-305a9a5e1a6d/connection-checker/0.log" Nov 03 04:56:55.037953 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:55.037876 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-d8k4b_0e4b8195-7644-4736-af66-d32d0d48551e/connection-checker/0.log" Nov 03 04:56:55.455407 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:55.455378 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nz8pk_83b404fd-1b9b-4088-ba59-d9997f5a1558/connection-checker/0.log" Nov 03 04:56:56.314364 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:56.314330 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jnblx_0613f917-8742-408c-80ed-640e913c05f6/manager/0.log" Nov 03 04:56:56.415504 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:56.415467 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qg9cd_724f0110-6fa5-473c-a88b-c514952a50a3/konnectivity-agent/0.log" Nov 03 04:56:57.960970 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:57.960930 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mn4ss_636bbe26-5388-49da-93d3-7d53a66b7d94/manager/0.log" Nov 03 04:56:58.061330 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:58.061299 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-120.us-west-1.compute.internal_143abffde7987a79bc89f8333af31943/haproxy/0.log" Nov 03 04:56:58.077986 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:56:58.077958 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pzvwf_9699142d-1bf4-46a4-9705-f5bd4c028ae6/proxy/0.log" Nov 03 04:57:00.011911 ip-10-0-138-120 kubenswrapper[5117]: E1103 04:57:00.011869 5117 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-762ml" containerName="extractor" Nov 03 04:57:02.248857 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.248821 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jnblx_0613f917-8742-408c-80ed-640e913c05f6/manager/0.log" Nov 03 04:57:02.888992 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.888954 5117 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8"] Nov 03 04:57:02.889435 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.889422 5117 cpu_manager.go:414] "containerMap: removing container" podUID="8671112d-396a-401d-a6be-71f0c61cccaf" containerName="maas-api" Nov 03 04:57:02.889506 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.889436 5117 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8671112d-396a-401d-a6be-71f0c61cccaf" containerName="maas-api" Nov 03 04:57:02.889565 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.889553 5117 memory_manager.go:365] "RemoveStaleState removing state" podUID="8671112d-396a-401d-a6be-71f0c61cccaf" containerName="maas-api" Nov 03 04:57:02.892750 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.892731 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:02.947825 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.947785 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ljm7\"/\"openshift-service-ca.crt\"" Nov 03 04:57:02.950515 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.950487 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8"] Nov 03 04:57:02.998550 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.998519 5117 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ljm7\"/\"kube-root-ca.crt\"" Nov 03 04:57:02.998550 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:02.998519 5117 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4ljm7\"/\"default-dockercfg-w89j4\"" Nov 03 04:57:03.027464 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.027415 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9szxj_70c7e0ef-af42-493c-baf8-1322dcbd8074/manager/0.log" Nov 03 04:57:03.039033 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.039001 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-proc\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.039199 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.039039 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbd8v\" (UniqueName: \"kubernetes.io/projected/57a1259f-0a76-4e59-aeb5-141ae5933062-kube-api-access-jbd8v\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.039199 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.039071 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-podres\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.039285 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.039202 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-lib-modules\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.039285 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.039275 5117 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-sys\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.140885 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.140778 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-sys\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.140885 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.140841 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-proc\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.140885 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.140870 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbd8v\" (UniqueName: \"kubernetes.io/projected/57a1259f-0a76-4e59-aeb5-141ae5933062-kube-api-access-jbd8v\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.141170 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.140902 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-sys\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.141170 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.140912 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-proc\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.141170 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.140915 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-podres\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.141170 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.140965 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-podres\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.141170 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.141030 5117 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-lib-modules\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.141170 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.141123 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57a1259f-0a76-4e59-aeb5-141ae5933062-lib-modules\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.202931 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.202889 5117 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbd8v\" (UniqueName: \"kubernetes.io/projected/57a1259f-0a76-4e59-aeb5-141ae5933062-kube-api-access-jbd8v\") pod \"perf-node-gather-daemonset-76bt8\" (UID: \"57a1259f-0a76-4e59-aeb5-141ae5933062\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.205653 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.205632 5117 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:03.592657 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:03.592597 5117 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8"] Nov 03 04:57:03.595700 ip-10-0-138-120 kubenswrapper[5117]: W1103 04:57:03.595671 5117 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod57a1259f_0a76_4e59_aeb5_141ae5933062.slice/crio-94481878e0dbe0f6e5a2db260fc608c536b065a8002b35363d3ed4f22f3a7d9a WatchSource:0}: Error finding container 94481878e0dbe0f6e5a2db260fc608c536b065a8002b35363d3ed4f22f3a7d9a: Status 404 returned error can't find the container with id 94481878e0dbe0f6e5a2db260fc608c536b065a8002b35363d3ed4f22f3a7d9a Nov 03 04:57:04.150780 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:04.150723 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mn4ss_636bbe26-5388-49da-93d3-7d53a66b7d94/manager/0.log" Nov 03 04:57:04.315827 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:04.315798 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pzvwf_9699142d-1bf4-46a4-9705-f5bd4c028ae6/proxy/0.log" Nov 03 04:57:04.576357 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:04.576319 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" event={"ID":"57a1259f-0a76-4e59-aeb5-141ae5933062","Type":"ContainerStarted","Data":"d94969385095253380cc30b583856a1e5991a6d1fc69a6569b7e414ceb618c92"} Nov 03 04:57:04.576567 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:04.576411 5117 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" event={"ID":"57a1259f-0a76-4e59-aeb5-141ae5933062","Type":"ContainerStarted","Data":"94481878e0dbe0f6e5a2db260fc608c536b065a8002b35363d3ed4f22f3a7d9a"} Nov 03 04:57:04.576567 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:04.576460 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:04.694792 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:04.694738 5117 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" podStartSLOduration=2.694723537 podStartE2EDuration="2.694723537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 04:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:57:04.693752808 +0000 UTC m=+6959.724351347" watchObservedRunningTime="2026-11-03 04:57:04.694723537 +0000 UTC m=+6959.725322104" Nov 03 04:57:08.290054 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:08.289967 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jnblx_0613f917-8742-408c-80ed-640e913c05f6/manager/0.log" Nov 03 04:57:09.952567 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:09.952528 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mn4ss_636bbe26-5388-49da-93d3-7d53a66b7d94/manager/0.log" Nov 03 04:57:10.118144 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:10.118115 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-pzvwf_9699142d-1bf4-46a4-9705-f5bd4c028ae6/proxy/0.log" Nov 03 04:57:10.592579 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:10.592552 5117 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-76bt8" Nov 03 04:57:10.684037 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:10.683992 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-p7hdq_fef674c6-31e4-4f47-b723-19bc2b38502c/cluster-monitoring-operator/0.log" Nov 03 04:57:14.421823 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:14.421791 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbt5z_20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5/node-exporter/0.log" Nov 03 04:57:14.835038 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:14.834990 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbt5z_20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5/kube-rbac-proxy/0.log" Nov 03 04:57:15.248966 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:15.248884 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbt5z_20e6a330-21fb-48f2-a98c-e0fcbeb8f2a5/init-textfile/0.log" Nov 03 04:57:15.622025 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:15.621995 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5cd8cf48-2vqdw_dfba0d42-1654-48c7-b958-e496bb4b94a3/kube-auth-proxy/0.log" Nov 03 04:57:16.492563 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:16.492535 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-9znf5_b15dddef-9e9b-4662-a65f-614203a4e981/istio-proxy/0.log" Nov 03 04:57:17.140063 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:17.140033 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-lbdzp_21acc925-025e-4dd2-bd7e-3314f7337a45/istio-proxy/0.log" Nov 03 04:57:17.783972 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:17.783933 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-rnx7l_3bc1ec57-9eb0-4153-a266-cc11de2854a2/istio-proxy/0.log" Nov 03 04:57:18.858071 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:18.858043 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c5c7c7bf7-x5tt5_c02e7385-46cc-46a6-822f-f47fe2819a3f/router/0.log" Nov 03 04:57:21.096525 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:21.096498 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2pcj8_48843ed4-9fc6-4073-8e2c-419df4be9f66/prometheus-operator/0.log" Nov 03 04:57:21.506488 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:21.506368 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2pcj8_48843ed4-9fc6-4073-8e2c-419df4be9f66/kube-rbac-proxy/0.log" Nov 03 04:57:21.932680 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:21.932631 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lk8lx_2eda3388-921b-4760-a85a-0726a5bd19a6/prometheus-operator-admission-webhook/0.log" Nov 03 04:57:24.248124 ip-10-0-138-120 kubenswrapper[5117]: I1103 04:57:24.248087 5117 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-9szxj_70c7e0ef-af42-493c-baf8-1322dcbd8074/manager/0.log"